ClaudeBot

What ClaudeBot does

ClaudeBot performs broad crawls of publicly available pages for Anthropic's model training. Anthropic documents the user agent and states that it honors robots.txt directives, including targeted disallow rules for the ClaudeBot token. Its crawl volume has drawn public complaints: in 2024 several site operators reported bursts of very heavy ClaudeBot traffic, and the episode became one of the standard examples in the debate about AI crawler load on independent sites. Anthropic has since emphasized rate limits and robots compliance in its documentation.

Why it visits your site

Like all training crawlers, it is harvesting text. Sites with deep archives see it walk old content as well as new posts, and reference-quality writing attracts repeat visits.

The bigger picture: AI training crawlers

Training crawlers are the most economically lopsided visitors your server has. A search crawler takes your content and pays you back in ranked listings and click-throughs; a training crawler takes the same content to improve a commercial model and, by design, sends nothing in return. The visit itself is the entire transaction. That asymmetry is why training bots - not AI in general - are where most of the blocking debate actually lives, and why the publishing industry's response has split between refusal and negotiation: several large publishers block these crawlers outright, while others have signed licensing deals with the same operators. Both strategies start from the same place - knowing, concretely, how much of your archive is being read and by whom.

The norms here are young and still hardening. The better operators now publish user agents, IP ranges, and opt-out tokens, and honor robots.txt; others publish nothing and, by widely credited reports, crawl through disallow rules. Treat each crawler on its record rather than assuming the category behaves uniformly - and remember that a robots.txt line only ever governs the polite end of the spectrum.

Should you block it?

The same calculus as GPTBot: no referral traffic comes back from a training crawl, so content businesses often block or meter it. Anthropic's user-initiated fetcher (Claude-User) and search indexer are controlled by separate tokens, so you can block training while staying reachable in Claude's live answers. If server load rather than principle is your concern, watching its actual request volume before deciding is more informative than assuming - on many sites it is modest, on some it is not.

Block via robots.txt

Add these lines to the robots.txt at your site root to ask ClaudeBot to stay away:

User-agent: ClaudeBot
Disallow: /

Remember that robots.txt is a request, not a lock - compliance is voluntary, and enforcement happens at the server. Viz's AI Control pairs robots.txt rules with server-level 403 responses and then probes your site to verify the block is actually holding.

How to see ClaudeBot traffic on your site

You have three windows onto it, each with a catch. Raw hosting access logs contain every ClaudeBot request, but many managed WordPress hosts don't expose them, and when they do you are grepping text files by hand. A CDN dashboard sees the traffic at the edge, but it lives outside WordPress and speaks in totals, not in your site's terms. And JavaScript analytics - GA4 and friends - will never show it at all, because ClaudeBot doesn't run scripts.

Viz's answer is the request log: filter to ClaudeBot and read exactly which URLs it fetched and when, right inside wp-admin - then watch the same filter after any blocking decision to see whether the visits stopped, slowed, or kept coming.