Bot directory

Every bot knocking on your WordPress site, named.

66 crawlers and fetchers, documented: who operates them, what they take, whether they honor robots.txt, and how to block the ones you don't want.

Every request to your server carries a user-agent string, and the legitimate crawler operators - OpenAI, Anthropic, Google, Perplexity, Meta - publish the names their bots announce. Those names matter because they are the handles you control crawlers with: a robots.txt rule addresses a bot by its token, and a server-level block refuses it by name. Know the name, and you can meter, allow, or refuse each visitor individually instead of guessing at “bot traffic” in aggregate.

Identity is layered, and the layers are where most blocking mistakes happen. One company often runs several bots with different jobs and different deals for you: OpenAI's training crawler, its search indexer, and its user-triggered fetcher are three different names with three different value exchanges attached. Blocking “OpenAI” is three separate decisions - and the operators split their tokens precisely so you can make them separately. The most common own-goal in AI-era bot policy is the blanket rule that blocks a company's search indexer or user-fetcher to stop its training crawler.

A decision about one category tells you nothing about another. Training crawlers take content and send nothing back; AI search bots index you so AI assistants can cite and link you; user agents fetch a page because one human asked about it right now; SEO tool crawlers harvest your link graph into commercial databases your own team may subscribe to. That is why this directory is organized by category before anything else.

User agents are self-reported. A polite bot tells the truth; an impolite one wears someone else's name or none at all, and the best-documented operators publish IP ranges precisely so their names can be verified. That is why serious bot control pairs name-based rules with server-side measurement - which is exactly what Viz does inside WordPress.

AI training crawlers

AI training crawler

Bulk crawlers that read your archive to build training corpora for AI models. They take content once and send nothing back.

GPTBot
OpenAI · OpenAI's web crawler for collecting training data for its foundation models, including the GPT series.
GPTBot
ClaudeBot
Anthropic · Anthropic's crawler for collecting public web data used to train the Claude family of models.
ClaudeBot
CCBot
Common Crawl · The crawler behind Common Crawl, a nonprofit archive of the web whose datasets are a staple ingredient of AI training corpora.
CCBot
Bytespider
ByteDance · ByteDance's aggressive web crawler, widely associated with AI training data collection and widely reported to ignore robots.txt.
Bytespider
Meta-ExternalAgent
Meta · Meta's crawler for training AI models and improving its products, introduced in 2024 as a dedicated, blockable token.
meta-externalagent
FacebookBot
Meta · An older Meta crawler documented as collecting public data to train AI models, including speech recognition systems.
FacebookBot
Amazonbot
Amazon · Amazon's web crawler, documented as improving Alexa's ability to answer questions and supporting other Amazon services.
Amazonbot
Applebot-Extended
Apple · Not a separate crawler: a robots.txt control token that tells Apple not to use content Applebot has crawled for training its foundation models.
Applebot-Extended
Google-Extended
Google · Not a separate crawler: a robots.txt control token that stops Google using your content for Gemini training and AI grounding, without touching Search.
Google-Extended
ImagesiftBot
The Hive (Hive AI) · An image-focused crawler operated by Hive, collecting publicly available images at scale; commonly associated with training visual AI models.
ImagesiftBot
Diffbot
Diffbot Technologies · A commercial extraction crawler that turns web pages into structured data and feeds Diffbot's machine-built Knowledge Graph.
Diffbot
Cohere-AI
Cohere · Robots.txt tokens associated with Cohere's training-data collection for its enterprise language models.
cohere-training-data-crawler
AI2Bot
Allen Institute for AI · The Allen Institute for AI's crawler, collecting web content to train its openly released language models and datasets.
AI2Bot
img2dataset
· An open-source tool that bulk-downloads images from URL lists into training datasets - anyone can run it, so behavior varies with the operator.
img2dataset
Anthropic-AI
Anthropic · A legacy token that appears in many AI blocklists; Anthropic's documented crawler is ClaudeBot, and this token's actual status is unclear.
anthropic-ai
iAskSpider
iAsk.AI · The crawler for iAsk.AI, an AI answer engine; community trackers report it does not respect robots.txt.
iaskspider
FriendlyCrawler
Unknown (self-described) · An anonymous crawler that describes itself as gathering data for machine learning experiments; nobody has ever claimed it.
FriendlyCrawler
Timpibot
Timpi · The crawler for Timpi, a decentralized search index whose data is also offered for AI training.
Timpibot
VelenPublicWebCrawler
Hunter.io · A business-data crawler operated for Hunter.io, feeding the datasets behind its contact-finding tools.
VelenPublicWebCrawler
Webzio-Extended
Webz.io · Webz.io's opt-out token governing whether its crawled web data can be resold for AI use.
Webzio-Extended
Omgili
Webz.io · Webz.io's long-running crawler token, dating back to the Omgili discussion-search engine and still seen collecting content for resold data feeds.
omgili
Kangaroo Bot
Kangaroo LLM · The crawler for Kangaroo LLM, an Australian project building an open-source model trained on Australian web content.
Kangaroo Bot
Sidetrade indexer bot
Sidetrade · Sidetrade's crawler, gathering company data to train the AI behind its order-to-cash software.
Sidetrade indexer bot
AI autonomous agents

AI autonomous agent

Agents that drive a real browser to complete tasks a person delegated - comparing, booking, form-filling.

Search engine crawlers

Search engine crawler

The classic index builders. Blocking these costs you organic search traffic - treat them differently from AI crawlers.

Googlebot
Google · Google's primary search crawler - the one bot almost no site can afford to block.
Googlebot
GoogleOther
Google · Google's generic crawler for internal research and product development, split out so it doesn't wear Googlebot's badge.
GoogleOther
Bingbot
Microsoft · Microsoft's search crawler - and, indirectly, a data source for Copilot and Bing's AI answers.
Bingbot
Applebot
Apple · Apple's crawler powering Siri suggestions, Spotlight, and Safari search features across a billion-plus devices.
Applebot
DuckDuckBot
DuckDuckGo · DuckDuckGo's modest crawler, supplementing the licensed indexes behind its privacy-focused search.
DuckDuckBot
YandexBot
Yandex · The crawler for Yandex, the dominant search engine in Russia and several neighboring markets.
YandexBot
PetalBot
Huawei (Petal Search) · The crawler for Huawei's Petal Search, the default search on Huawei devices without Google services.
PetalBot
Baiduspider
Baidu · The crawler for Baidu, the dominant search engine in China.
Baiduspider
Yahoo! Slurp
Yahoo · Yahoo's historic search crawler, now a minor player since Yahoo's results have been powered by Bing for years.
Slurp
Sogou Spider
Sogou (Tencent) · The crawler for Sogou, a Chinese search engine owned by Tencent, with a long-standing reputation for heavy-handed crawling.
Sogou web spider
Exabot
Exalead (Dassault Systèmes) · The crawler for Exalead, a French search company acquired by Dassault Systèmes - largely a historical artifact today.
Exabot
SeznamBot
Seznam.cz · The crawler for Seznam.cz, the homegrown Czech search engine that still holds meaningful share against Google in its market.
SeznamBot
Utility & developer tools

Utility & developer tool

HTTP libraries, monitoring services, and platform fetchers. The user agent names the tool that made the request, not the person running it.

See them on your site

This directory tells you who they are. Viz tells you who's here.