Google-Extended
Not a separate crawler: a robots.txt control token that stops Google using your content for Gemini training and AI grounding, without touching Search.
Official documentation: developers.google.com
What Google-Extended does
Google-Extended has no user agent and makes no requests. Googlebot continues to crawl normally; disallowing this token tells Google not to use the crawled content to train Gemini models or as grounding for AI answers. Your search indexing and ranking are unaffected, per Google's documentation. Google introduced the token in September 2023 after sustained publisher pressure for a way to refuse AI training without leaving Search - an implicit acknowledgment that the two uses of a crawl are different deals.
Why it visits your site
It does not visit - it is a policy switch read from robots.txt, honored by Google's existing crawl infrastructure. You will never see it in a request log; the only place it exists is your robots.txt file.
The bigger picture: AI training crawlers
Training crawlers are the most economically lopsided visitors your server has. A search crawler takes your content and pays you back in ranked listings and click-throughs; a training crawler takes the same content to improve a commercial model and, by design, sends nothing in return. The visit itself is the entire transaction. That asymmetry is why training bots - not AI in general - are where most of the blocking debate actually lives, and why the publishing industry's response has split between refusal and negotiation: several large publishers block these crawlers outright, while others have signed licensing deals with the same operators. Both strategies start from the same place - knowing, concretely, how much of your archive is being read and by whom.
The norms here are young and still hardening. The better operators now publish user agents, IP ranges, and opt-out tokens, and honor robots.txt; others publish nothing and, by widely credited reports, crawl through disallow rules. Treat each crawler on its record rather than assuming the category behaves uniformly - and remember that a robots.txt line only ever governs the polite end of the spectrum.
Should you block it?
This is the rare opt-out with no stated search cost, which is why a large share of top news sites adopted it within months of launch. Note the limits: it does not remove you from AI Overviews built on Search - Google treats those as a Search feature governed by ordinary indexing controls - and it only binds Google. Setting it is close to free; just be clear-eyed that it narrows one use of your content rather than ending AI exposure.
Opt out via robots.txt
Add these lines to the robots.txt at your site root. Google-Extended is a policy token: the operator reads this rule and applies it to content its regular crawler has already fetched.
User-agent: Google-Extended Disallow: /
How to see Google-Extended traffic on your site
You won't - Google-Extended never appears in any log, because it never makes a request. To understand what it governs, watch the operator's regular crawler instead: its visits are the content this token's policy applies to. The meaningful check is your robots.txt file itself - confirm the disallow line is present and spelled exactly right, because a policy token with a typo governs nothing.