Somewhere right now a mid-size content site is paying more to serve AI training crawlers than to serve its actual readers. Bots passed humans on the open web this year, and the crawlers that took over do not send traffic back the way search engines did. The new development is not the scale — it is that a price has now been wired directly into the web's plumbing, using an HTTP status code that has sat dormant since 1996.
A price tag, wired into HTTP itself.
The mechanism reuses HTTP 402 “Payment Required” to charge AI crawlers per fetch, with signed request headers proving who is paying. It moves the crawler question from block-or-allow to block, allow, or charge — and because it rides open web standards, it could become a habit far beyond any single provider.
The crossover that changed the math
For most of the web's history, crawlers were polite and worth tolerating: search engines honoured robots directives, crawled gently, and sent readers back in exchange. The AI wave broke that bargain. Training crawlers want your entire archive; retrieval bots hit you in real time every time a chatbot cites you; and a long tail of anonymous scrapers ignores robots entirely and rotates through residential proxies when blocked. The old gentleman's agreement is gone, and treating robots.txt as a control rather than a suggestion is how origins melt.
The tipping point was measurable. In mid-2026, network telemetry showed automated requests crossed the majority line — bots now generate more than half of all requests to HTML content, and a large share of that is AI training traffic. The imbalance shows up most starkly in what some providers call the crawl-to-refer ratio: some AI crawlers fetch thousands, even tens of thousands, of pages for every single visitor they send back. That is the number that turned a tolerance problem into an economics problem.
What it does to your infrastructure
The damage lands in three places. Egress: archive crawls pull long-tail content that was never in cache, so it comes off your origin and your bill. Compute: retrieval bots love the most expensive URLs you own — search endpoints and faceted category pages that are uncacheable by design. And analytics pollution: if half your “users” are machines, your conversion data and capacity planning are fiction.
The cache point is the quiet killer. Human traffic follows popularity curves that caches love; crawlers walk the whole tree uniformly. A site with a 95% hit ratio for humans can watch it collapse into the seventies during a hard crawl — the CDN bill stays flat while the origin bill does not. That is why the crawl shows up first in your origin hit ratio, before it shows up anywhere else.
How pay-per-crawl works
The mechanism is elegant because it reuses what already exists. When an AI crawler requests a page from a participating site without payment, the edge returns HTTP 402 “Payment Required” with pricing information attached. The crawler can retry with a signed request — cryptographically signed headers that prove which crawler is paying — and receive the content, charged per successful fetch. There is also a proactive flow where the crawler declares a maximum price up front, and if the site's price is at or below it, the content comes back on the first request. The provider sits in the middle as the merchant of record, aggregating the micropayments and paying publishers.
The 402 status code has existed in the HTTP specification since 1996 as a placeholder for exactly this kind of payment system; it simply never had an implementation at scale. Wiring it into edge infrastructure is what makes it real. And because the signalling rides open web standards rather than a proprietary API, other providers and agents can in principle speak the same protocol — which is what could turn one company's feature into a web-wide habit.
From blocking to a default, fast
This is moving quicker than most site owners realise. The framework has already shifted from “charge a bot to fetch a page” toward charging when the content actually creates value in an AI answer, and crawler permissions are being split into distinct categories — search, AI training, and AI agents — that a site can allow, block, or monetise independently. Providers have signalled that new and free-tier domains will begin defaulting to blocking AI training and agent crawlers on ad-supported pages unless the crawler pays. For a lot of small sites, that block-allow-or-charge decision now lives in a dashboard setting the owner has never opened.
The decision framework
Do not reach for a blanket rule; “block everything” and “charge everything” both leave money and visibility on the table. If you monetise attention, block training by default, allow verified search, and consider a toll for AI retrieval. If you monetise APIs, the bots may already be your customers — meter, do not block. If you are e-commerce, your real enemy is scraper-driven price intelligence, a different problem from crawler policy. And watch your origin hit ratio, because that is where the crawl announces itself before any invoice does. The right setting is per-URL, based on what each crawl is actually worth to you — content that gets you cited is often worth keeping open, because the citation is worth more than the fee.