The AI crawler debate has hardened into two lazy positions: "block everything, they're stealing your content" and "block nothing, you'll miss the AI wave." Both skip the only step that matters: knowing what is actually crawling you, at what volume, and what you get back. Blocking is not a philosophy. It is a per-bot business decision, and it is cheap to get right once you can see the traffic.
Start with your business model, not your feelings
The right posture depends almost entirely on how your content makes money:
- Content is the product (publishers, course sellers, paywalled research): training crawlers consume your inventory and return nothing attributable. You have the strongest case for blocking training bots broadly - and, at sufficient scale, for treating crawl access as something to license rather than give away.
- Content markets the product (SaaS, e-commerce, services): AI engines that read your docs and comparison pages can recommend you in answers your buyers are already asking. Broad blocking mostly hurts you. The interesting question becomes which pages retrieval bots value - that is your AI-era positioning, measured.
- Mixed models (most sites): split the difference by bot job - block training, allow retrieval and user agents - or by section, disallowing your premium paths while leaving marketing pages open.
Why the old crawl bargain broke
For twenty-five years, letting crawlers in was obviously correct, because the web ran on a stable exchange: search engines crawled your pages and paid you back in ranked visibility and clicks. Both sides profited, so a voluntary protocol - robots.txt - was enough to govern it. Nobody needed enforcement because nobody sane defected.
AI training changed the payout structure. A training crawler consumes the same pages but returns nothing attributable to you - the value surfaces later, inside a model, unlinked. Retrieval bots sit in between: some answers cite you and send readers, many do not. This is why the industry is fracturing into deals and disputes at the same time - some large publishers have signed licensing agreements with AI companies while others litigate - and why a norm that ran on mutual benefit now needs actual measurement and actual enforcement behind it. Your robots.txt went from etiquette everyone honored to an opening position in a negotiation.
robots.txt is a request, not a wall
The first tool everyone reaches for is robots.txt, and it is worth being precise about what it is: a published preference that well-behaved crawlers choose to honor. The major operators - OpenAI, Anthropic, Google, Apple - document their tokens and generally comply. A meaningful tail of crawlers ignores it entirely, and a few have been publicly accused of doing so while claiming otherwise.
# A common publisher posture: User-agent: GPTBot # block model training Disallow: / User-agent: CCBot # block open-archive training Disallow: / User-agent: Google-Extended # opt out of Gemini training Disallow: / User-agent: OAI-SearchBot # stay citable in ChatGPT answers Allow: /
Use robots.txt because it is the standard, honored by the actors who matter most, and legible to everyone. Just do not confuse publishing a preference with enforcing one.
Also remember that blocking is not all-or-nothing. Path rules let you refuse crawlers your premium inventory while leaving marketing pages open - often the most commercially sensible split:
User-agent: GPTBot Disallow: /guides/ Disallow: /research/ Allow: /
(One caution from the standard: Crawl-delay is not part of RFC 9309 and the major AI crawlers do not commit to honoring it - do not build a metering strategy on it.)
Enforcement means the server says no
If a bot ignores robots.txt, the next layer is refusing the request: matching the bot at the server and returning a 403 before your content is served. This is enforcement rather than etiquette - the crawler gets nothing regardless of its manners. The catch is verification: a blocked request today tells you nothing about next month unless something keeps checking. This is why Viz's AI Control pairs layered robots.txt rules and server-level 403s with a probe that verifies the block is actually holding over time.
Where to enforce: application or edge
Refusal can live at two layers, and they fail differently. Blocking at the application (inside WordPress) knows your site - which paths are premium, which are marketing - and keeps policy next to your content, in a dashboard you already use. Blocking at the edge (CDN or WAF bot rules) refuses requests before they cost you origin compute and can cover requests your cache would otherwise serve - but the rules live in another vendor's console, tend to be coarser, and drift out of sync with what you actually publish. Mature setups often use both: application-level policy as the source of truth, edge rules as the outer wall for the highest-volume offenders. Whichever you choose, the failure mode is the same - silent decay. Operators rotate user agents and infrastructure, caches get reconfigured, and a block that worked in March can leak by June. That is why verification has to be continuous, not a one-time test: it is exactly the job of the probe in Viz's AI Control.
The one mistake you cannot walk back quickly
Whatever you decide about AI, do not catch search engines in the blast radius. Blocking Googlebot or Bingbot deindexes you, and recovery is slow even after you fix it. Bing deserves particular care: its index feeds ChatGPT search and Copilot, so blocking Bingbot quietly removes you from AI answers you probably wanted. The safe pattern is explicit: name the training bots you refuse, and leave search crawlers untouched. If you want to refuse Google's AI training specifically, that is what Google-Extended is for - it costs you nothing in search.
Meter before you wall
There is a middle path between open door and locked gate that most commentary skips: measure first, decide per bot, revisit quarterly. Concretely:
- Watch a month of classified traffic. Volume changes minds - a bot hitting you 40 times a day is a different conversation than one hitting you 40,000 times.
- Compare crawl volume against what each operator sends back in referrals. A retrieval bot that cites you and drives readers is a partner; one that reads everything and sends nobody is a cost.
- Block the clear losses, allow the clear wins, and leave the rest measured. Positions age fast in this space - operators change behavior, launch new bots, and start (or stop) sending traffic.
What blocking changes - and what it cannot
Set expectations before you flip anything. Blocking a training crawler is forward-looking only: whatever was crawled before your block is already in someone's dataset, and no robots.txt line reaches back into a trained model. The payoff arrives on the next training run, for the content you publish from now on - which is precisely why the decision gets more valuable the earlier you make it and the more you publish. Blocking retrieval bots works on the opposite clock: the effect is fast and visible, because the engine can no longer fetch you at answer time, and you should expect your presence in that engine's citations to fade accordingly. Blocking user-agent fetches is the strangest of the three - the "bot" you refuse is often a person, mid-conversation, asking an assistant about your page right now. Each block is reversible in robots.txt terms and much slower to reverse in consequences, so match the layer you pull to the outcome you actually want.
From binary to metered
Block-or-allow is probably a transitional framing anyway. The direction of travel is toward metered access: licensing agreements between publishers and AI companies already exist at the top of the market, infrastructure vendors have begun experimenting with pay-per-crawl mechanics, and the long-dormant HTTP 402 ("Payment Required") status code keeps coming up in serious proposals. None of this is standardized yet, and small sites cannot negotiate like wire services. But the strategic implication is available to everyone today: whatever pricing or permission regime emerges, it will be operated from crawl records. Sites that can produce a dated, per-bot history of who consumed what will walk into that regime with leverage; sites that cannot will take whatever default they are offered. Measurement is how you keep your options open under rules that have not been written yet.
The waiting position is a position
One more option deserves a defense: deliberately deciding nothing yet, while measuring. Blocking has option value going one direction - you can always block later, and the content a training crawler already read is not coming back, but next quarter's content is still yours to withhold. Allowing has option value the other way: staying crawlable while answer engines are forming their picture of your niche may be cheap positioning you cannot buy back later. Measured waiting is only a trap when it is unmeasured - drifting with no data and calling it patience. If your crawl log is empty of insight a quarter from now, you have not been waiting; you have been ignoring.
A quarterly review makes the waiting productive. Fifteen minutes: check who the top five crawlers were, whether any new named bot appeared, whether your blocked list still matches your business reasoning, whether referral return from each answer engine moved, and whether any block has started leaking. That cadence matches how fast the ecosystem actually changes - faster than annual policy, slower than daily anxiety.
Special cases worth knowing about
- Heavily cached sites: a full-page cache can keep serving a bot you blocked at the origin, because the cached copy never asks WordPress for permission. If you block, verify the block from outside - do not assume.
- Staging and preview sites: the goal there is not selective blocking but total invisibility - authentication or, at minimum, a blanket disallow plus noindex. Training crawlers finding your half-finished redesign helps nobody.
- Licensing-scale publishers: if you may one day charge for crawl access, your request log is the asset. A dated record of who crawled what, at what volume, before and after your opt-outs, is the evidence a licensing conversation starts from.
Note what this framework requires: measurement you trust. Every branch of the decision - which bots, what volume, what referral return - is empirical, and none of it is visible in JavaScript analytics. You cannot negotiate, meter, or even meaningfully refuse what you cannot see. Whatever you choose to do about AI crawlers, the first move is the same: turn on the lights.