Cloudflare flips the default and starts charging crawlers
AI bots blocked unless allowed, plus a marketplace for per-crawl payment. A fifth of the web changes its robots policy at once.
Cloudflare announced today that new domains on its network will block AI crawlers by default, and launched a pay-per-crawl marketplace letting site operators charge for access.
Cloudflare sits in front of roughly a fifth of the web. A default change at that position is not a product launch, it is a policy change for the internet.
the mechanism#
Two pieces.
Default blocking. New zones get AI crawler blocking on unless the operator opts out. Cloudflare maintains the bot classification — separating search crawlers, which drive traffic back, from training crawlers, which do not.
Pay-per-crawl. A site sets a price. A crawler that wants the content gets an HTTP 402 Payment Required with terms. Cloudflare handles settlement.
HTTP 402 has been "reserved for future use" since 1997. It is genuinely funny that this is what activated it.
why the old system failed#
robots.txt is a request, not a control. It works because well-behaved crawlers choose to honor it, and that consensus held for thirty years because search engines had an incentive to be well-behaved — they needed publishers to not block them.
AI training crawlers have no such incentive. The content is valuable to them and the traffic they return is zero or nearly so. Multiple studies found training crawlers ignoring robots.txt, rotating user agents, and using residential proxy pools. Once a norm has no enforcement and no incentive, it stops being a norm.
Cloudflare's move replaces a request with a control. That is the actual innovation and it required no new technology at all — just someone at a chokepoint deciding to enforce.
the case against#
Concentrating the ability to gate the web at one CDN is not obviously good, even if this specific use of the power is popular.
The precedent is: an infrastructure company can unilaterally change how content is accessed for a large fraction of the internet, and the mechanism generalizes to things other than AI crawlers. Cloudflare has been thoughtful and has taken public positions on not being an arbiter, and the concentration is still real.
There is also a smaller-player problem. Large AI companies can negotiate licensing deals directly. Researchers, startups, the Internet Archive, and academic crawlers cannot. A tollbooth is regressive: it is a rounding error for the incumbents and a barrier for everyone else. Cloudflare has carve-outs for some of these and the carve-outs are discretionary, which is the point.
for developers#
Two practical items.
If you run a site, decide deliberately. Blocking training crawlers is now the default; that may not be what you want. Documentation sites in particular may prefer to be in the training data, because being the thing the model knows about is worth more than the pageview you lost.
If you build anything that crawls, expect 402s and expect your user agent to matter. Identify honestly, respect the directives, and set up billing if you need paid access. The era of scraping quietly is ending, and the enforcement is technical now rather than legal.
the bigger shift#
The web's economic model was: publish freely, get traffic, monetize traffic. AI answers break the second step, which breaks the third, which will eventually break the first.
Pay-per-crawl is one proposed replacement. Licensing deals are another. Neither is obviously going to work at the scale of the actual web, where most content is made by people with no ability to negotiate anything.
What replaces it is genuinely unresolved, and 2025 is the year everybody stopped pretending otherwise.
— Dom, July 1, 2025