"AI bots are crawling my site" sounds like one phenomenon. It is at least three, run by different systems with different economics, and a site owner who treats them as one blob will make bad decisions about all of them. This guide sorts the major crawlers you'll actually see in your logs into their three real jobs: training, retrieval, and acting for a user.
The three jobs an AI bot can have
- Training crawlers read your archive to build the dataset a future model learns from. They crawl broadly, favor text, and the value exchange is deferred and diffuse: your content improves a model that millions use, and you get nothing directly attributable.
- Retrieval (search) bots fetch pages at question time so an AI engine can ground its answer in current content. They crawl narrowly and repeatedly, hitting the pages the engine considers authoritative. Some answers cite you and send readers.
- User agents fetch a page because a specific human asked an assistant to read it - "summarize this article" - or clicked something an assistant surfaced. This is the closest to a human visit and blocking it usually annoys a real person.
The same company typically runs all three, under different names, with different robots.txt tokens - which is exactly why the distinction is actionable. You can welcome one and refuse another.
A short history of the tokens
robots.txt predates every company in this guide: it emerged in 1994 as a voluntary convention among early crawler operators and was only formalized as a standard (RFC 9309) in 2022. For nearly three decades one token per crawler was enough, because crawlers mostly did one job - indexing. The AI era broke that assumption. Operators now expose purpose-specific tokens precisely so sites can say yes to one use of their content and no to another: Google separated Google-Extended from Googlebot, Apple added Applebot-Extended alongside Applebot, OpenAI split its fleet three ways. Reading a modern robots.txt is reading a menu of consent choices that did not exist a few years ago - and the vocabulary is still growing.
Trust, but verify the name
A user-agent string is a claim, not a credential - anything can call itself Googlebot. The major operators know this and publish verification paths: Google, Microsoft, and OpenAI all document official IP ranges or support reverse-DNS checks, so a request claiming to be their bot can be confirmed or exposed. Two practical consequences for site owners: first, a "Googlebot" hitting you from a residential IP is not Googlebot, and treating impostors as the real thing corrupts every decision downstream. Second, verification is what makes enforcement safe - you can 403 a fake Googlebot with zero SEO risk once you know it is fake.
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
OpenAI splits its fleet cleanly along the three jobs. GPTBot is the training crawler; disallowing it is how you opt your content out of future model training. OAI-SearchBot powers ChatGPT's search index, and allowing it is what makes you eligible to be cited (with links) in ChatGPT answers. ChatGPT-User fires when a user's session fetches your page live. Each honors its own robots token:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
That example - block training, allow search and user fetches - is a coherent position many publishers land on: no free training data, but stay visible where answers cite sources.
Anthropic: ClaudeBot, Claude-User
ClaudeBot is Anthropic's main crawler, gathering content that can inform Claude's training. Claude-User covers fetches made on behalf of a person using Claude. Anthropic documents an anthropic-ai robots token as well; honoring the specific bot names in robots.txt is the safest way to express intent. ClaudeBot is generally well-behaved about robots.txt and identifies itself with a contact address in its user agent.
Perplexity: PerplexityBot, Perplexity-User
Perplexity is an answer engine, so its crawling is retrieval-heavy. PerplexityBot builds and refreshes its index; Perplexity-User fetches when a user's query needs your page right now. Perplexity answers cite sources prominently, which makes it one of the more measurable AI referral channels - you can see the crawl on your server and the clicks in your referral reports, and judge the exchange rate yourself.
Google: Googlebot vs Google-Extended
Google's split matters more than any other, because getting it wrong can hurt your search traffic. Googlebot is search indexing - block it and you disappear from Google. Google-Extended is not a separate crawler at all; it is a robots.txt control that tells Google not to use content Googlebot already fetched for Gemini training and grounding. Disallowing Google-Extended costs you nothing in search. Blocking Googlebot to "stop AI" is the classic self-inflicted wound.
Microsoft: the quiet kingmaker
Bingbot deserves more attention than it gets in AI conversations. Bing's index does double duty: it powers Bing search, and it grounds Microsoft Copilot and feeds ChatGPT's search experience. That makes Bingbot one of the highest-leverage crawlers on your site - blocking it does not just remove you from a search engine with modest market share; it quietly removes you from a family of AI answer surfaces. The general lesson: an index crawled once is now consumed by many downstream products, so evaluate a search crawler by everything its index feeds, not by the search engine's brand alone.
The volume kings you didn't invite
Your logs will also show crawlers whose value exchange is murkier. CCBot (Common Crawl) feeds an open web archive that many model builders train on - blocking it is a broad opt-out with one rule. Bytespider (ByteDance) has a reputation for aggressive crawl volume and inconsistent robots.txt behavior. Amazonbot and Applebot-Extended (Apple's training opt-out control, distinct from Applebot proper, which powers Siri and Spotlight suggestions) round out the fleet, and Meta-ExternalAgent covers Meta's AI data collection.
The long tail: everything that isn't a household name
Beyond the majors, expect a persistent drizzle of smaller visitors: research crawlers from universities, open-source scraping frameworks with default user agents, SEO tools (AhrefsBot, SemrushBot) mapping your backlinks, archive projects, and one-off scripts someone wrote on a laptop. Individually negligible, collectively they are often a double-digit share of automated traffic. The sane posture is triage: identify the named ones and decide deliberately, watch the unnamed remainder as a trend line, and investigate only when a single unknown source starts pulling meaningful volume. Chasing every curl request is a waste of your attention; missing a new crawler that suddenly reads 10,000 pages is not.
A sane default policy, written out
Policies differ by business model, but if you want a defensible starting point that most content-led sites can adapt, it looks like this:
# Training: opt out of the big archives User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / # Retrieval and user fetches: stay citable User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ChatGPT-User Allow: / # Search: never touch User-agent: Googlebot Allow: / User-agent: Bingbot Allow: /
Read the shape of it: refuse deferred, unattributable uses (training), allow uses that can cite and send readers (retrieval, user fetches), and leave classic search untouched. Then treat it as a living document - review it against your actual crawl data quarterly, because operators add tokens and change behavior, and a policy written once and forgotten drifts out of sync with reality within months.
The fourth job: agents that browse like people
A newer category deserves its own entry in the field guide, because it breaks the neat pattern above. Agentic AI - browser-driving assistants and autonomous task frameworks - does not fetch your HTML and leave. It operates a real browser: it executes JavaScript, clicks, fills forms, follows redirects. Sometimes it announces itself with an agent-specific user agent; often it looks like an ordinary browser session with slightly wrong rhythms - too fast between pages, too methodical, active at 4 a.m.
Agents scramble the measurement map in both directions. They can appear in your JavaScript analytics looking human, quietly inflating the numbers every other bot deflates. And they blur the policy question: an agent booking a table or comparing your pricing on behalf of a real customer is commercially closer to a human visit than to a crawl, so reflexively blocking everything automated will eventually mean turning away customers who happen to arrive by proxy. There is no settled best practice here yet. The honest position is to watch the category separately - which is why Viz gives autonomous agents their own bucket instead of folding them into "bots" - and revisit as conventions form.
Reading your own logs like a field guide
Once you can see this traffic classified, patterns emerge quickly. Training crawlers sweep your archive - old posts, tag pages, everything. Retrieval bots concentrate on a handful of pages, and that concentration is information: it is a map of what AI engines consider you authoritative on. A spike from a single bot often means a new model vendor just discovered you. And a steady drumbeat of ChatGPT-User or Perplexity-User hits on one article means real humans are asking assistants about the thing that article covers.
The shapes are distinctive enough that you can often name the job before you read the user agent. A training sweep is broad and shallow - thousands of distinct URLs, one visit each, marching through your archive in something close to sitemap order. Retrieval is narrow and deep - the same twenty URLs, refreshed on a cadence, biased toward whatever your niche's current questions are. User-agent fetches are single pages with no neighbors, arriving at human-irregular times, because each one is a person somewhere asking about that page right now. Cadence, breadth, and repetition tell the story; the user-agent string just signs it.
The one-glance recap
- Training: GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent - broad archive sweeps, deferred value, the strongest case-by-case blocking candidates.
- Retrieval: OAI-SearchBot, PerplexityBot, DuckAssistBot - narrow, repeated fetches; blocking trades citation visibility for privacy of your content at answer time.
- User fetches: ChatGPT-User, Claude-User, Perplexity-User - one page, one human, right now; block only with a reason.
- Control tokens, not crawlers: Google-Extended, Applebot-Extended - robots.txt switches governing AI use of content fetched by Googlebot and Applebot; free to disallow, no search risk.
- Never block casually: Googlebot, Bingbot - search visibility, and in Bingbot's case a growing share of AI answer surfaces, ride on them.
We keep a fuller reference - one page per bot, with its user-agent strings, robots token, and balanced blocking guidance - in the Viz bot directory. And if you want to see which of these are already hitting your own WordPress site, that is precisely what Viz's free plan shows you, classified into these categories from the first request.