Applebot-Extended
Not a separate crawler: a robots.txt control token that tells Apple not to use content Applebot has crawled for training its foundation models.
Official documentation: support.apple.com
What Applebot-Extended does
Applebot-Extended never fetches pages itself. Apple's regular Applebot does the crawling; this token is a documented switch that removes your content from Apple Intelligence model training while leaving Siri and Spotlight search features untouched. Apple introduced it in 2024 as Apple Intelligence launched, following the same crawl-once, control-uses-separately pattern Google set with Google-Extended - and reporting at the time found many large publishers opted out within months.
Why it visits your site
It does not visit. You will never see it in server logs - it exists only as a line in robots.txt that Apple reads and applies to content its regular crawler already fetched.
The bigger picture: AI training crawlers
Training crawlers are the most economically lopsided visitors your server has. A search crawler takes your content and pays you back in ranked listings and click-throughs; a training crawler takes the same content to improve a commercial model and, by design, sends nothing in return. The visit itself is the entire transaction. That asymmetry is why training bots - not AI in general - are where most of the blocking debate actually lives, and why the publishing industry's response has split between refusal and negotiation: several large publishers block these crawlers outright, while others have signed licensing deals with the same operators. Both strategies start from the same place - knowing, concretely, how much of your archive is being read and by whom.
The norms here are young and still hardening. The better operators now publish user agents, IP ranges, and opt-out tokens, and honor robots.txt; others publish nothing and, by widely credited reports, crawl through disallow rules. Treat each crawler on its record rather than assuming the category behaves uniformly - and remember that a robots.txt line only ever governs the polite end of the spectrum.
Should you block it?
Disallowing Applebot-Extended is close to free: you opt out of model training while keeping whatever visibility Applebot's search features give you. Apple offers no search referral upside tied to training consent, so there is no bargain to weigh. If you have decided against donating training data, set it.
Opt out via robots.txt
Add these lines to the robots.txt at your site root. Applebot-Extended is a policy token: the operator reads this rule and applies it to content its regular crawler has already fetched.
User-agent: Applebot-Extended Disallow: /
How to see Applebot-Extended traffic on your site
You won't - Applebot-Extended never appears in any log, because it never makes a request. To understand what it governs, watch the operator's regular crawler instead: its visits are the content this token's policy applies to. The meaningful check is your robots.txt file itself - confirm the disallow line is present and spelled exactly right, because a policy token with a typo governs nothing.