GPTBot

What GPTBot does

GPTBot crawls publicly accessible pages and feeds them into the corpora OpenAI uses to train and improve its models. It is a bulk crawler: it reads broadly and repeatedly across your archive rather than fetching single pages on demand. OpenAI publishes its user agent, its IP ranges, and an opt-out mechanism - which makes GPTBot one of the few AI crawlers whose identity you can actually verify, since a request claiming to be GPTBot from an unlisted IP is an impostor. Introduced in August 2023, it was the first major AI training crawler to ship with a documented opt-out, and it set the template the rest of the industry copied.

Why it visits your site

Your content is useful training material. Text-heavy, well-structured WordPress sites - blogs, documentation, publications - are exactly what model training pipelines want. Expect it to walk deep archives, not just recent posts: to a training pipeline, a well-written page from 2016 is worth as much as one from last week.

The bigger picture: AI training crawlers

Training crawlers are the most economically lopsided visitors your server has. A search crawler takes your content and pays you back in ranked listings and click-throughs; a training crawler takes the same content to improve a commercial model and, by design, sends nothing in return. The visit itself is the entire transaction. That asymmetry is why training bots - not AI in general - are where most of the blocking debate actually lives, and why the publishing industry's response has split between refusal and negotiation: several large publishers block these crawlers outright, while others have signed licensing deals with the same operators. Both strategies start from the same place - knowing, concretely, how much of your archive is being read and by whom.

The norms here are young and still hardening. The better operators now publish user agents, IP ranges, and opt-out tokens, and honor robots.txt; others publish nothing and, by widely credited reports, crawl through disallow rules. Treat each crawler on its record rather than assuming the category behaves uniformly - and remember that a robots.txt line only ever governs the polite end of the spectrum.

Should you block it?

If your content is your product, blocking or metering GPTBot is a defensible choice: training crawls send you no traffic in return. Within months of its launch, a large share of major news publishers had disallowed it - and some of those same publishers later signed licensing deals with OpenAI, which is the point worth absorbing: a block is not just protection, it is a negotiating position. The trade-off is speculative rather than immediate - being in training data may make models more likely to know your brand. Blocking GPTBot does not affect ChatGPT search results, which use OAI-SearchBot instead.

Block via robots.txt

Add these lines to the robots.txt at your site root to ask GPTBot to stay away:

User-agent: GPTBot
Disallow: /

Remember that robots.txt is a request, not a lock - compliance is voluntary, and enforcement happens at the server. Viz's AI Control pairs robots.txt rules with server-level 403 responses and then probes your site to verify the block is actually holding.

How to see GPTBot traffic on your site

You have three windows onto it, each with a catch. Raw hosting access logs contain every GPTBot request, but many managed WordPress hosts don't expose them, and when they do you are grepping text files by hand. A CDN dashboard sees the traffic at the edge, but it lives outside WordPress and speaks in totals, not in your site's terms. And JavaScript analytics - GA4 and friends - will never show it at all, because GPTBot doesn't run scripts.

Viz's answer is the request log: filter to GPTBot and read exactly which URLs it fetched and when, right inside wp-admin - then watch the same filter after any blocking decision to see whether the visits stopped, slowed, or kept coming.