AI crawler traffic is no longer a curiosity in the logs; on many sites it is the largest single class of non-human requests, and blocking it has become an infrastructure decision with a revenue consequence attached. The 2026 shift is that “block AI bots” is now the wrong shape of question. Crawlers do three different things with your content, and each deserves a different answer.
Sort the traffic into three intents
The useful taxonomy in 2026 splits automated AI traffic by what it does with what it takes. Search crawlers index your content so a product can answer questions about it later, usually with a link back — that is a referral channel and behaves like classic search. Agent traffic acts in real time on behalf of a person: a chat client fetching a page a user asked about, or a browser-driving agent completing a task. Training crawlers absorb the content into model weights, with no link, no visit and no recurring relationship.
Those three have genuinely different value. Blocking search removes you from a discovery channel. Blocking agents cuts off a class of visitor that increasingly arrives with purchase intent but never renders your page the way a human would. Blocking training protects content from permanent absorption but returns nothing in traffic. Any policy that treats them as one category is either giving away more than you meant to or losing traffic you wanted.
The trap worth knowing about before you set anything: several major crawlers are multi-purpose, combining search and training behind one identifier. Where policy is evaluated per intent and the most restrictive applicable rule wins, blocking training also blocks the general-purpose crawlers that carry your search visibility. Cloudflare’s July 2026 controls make this explicit, and its September 2026 default change applies training and agent blocks to ad-displaying pages on new domains and existing free-tier zones — which means a policy nobody consciously set can remove a site from classic search. If you run behind that platform, check the setting before the date rather than after your search console tells you.
Why robots.txt cannot enforce any of it
robots.txt expresses a preference to well-behaved clients. It has no enforcement mechanism, no authentication and no way to distinguish a compliant crawler from something wearing its user agent. Its value is real but narrow: it documents intent, it is honoured by the operators who publish their agents and IP ranges, and it gives you a defensible written record of what you asked for. It will not stop the traffic that ignores it, and a meaningful share of AI-classified traffic does.
The newer descriptive file conventions have the same limitation with less of the benefit. A markdown map of your site describes what you are, which is helpful for navigation, but it cannot restrict anything, and adoption among the model providers it was aimed at has been thin — major search guidance in 2026 explicitly declined to treat it as a ranking or access signal. Publish one if it helps your own documentation; do not put access control weight on it.
The practical division of labour: text files declare policy, the edge enforces it. Keep both, and keep them consistent — a robots.txt that permits what your WAF blocks produces confusing 403s for cooperative crawlers and tells the uncooperative ones nothing at all.
Enforce at the edge: identity before rules
Enforcement begins with identity, because a rule matched on a spoofable string is not a control. Verify a claimed crawler before you act on the claim: check that the source address belongs to the published range for that operator, or that reverse and forward DNS agree, or — increasingly — that the request carries a cryptographic signature. Signed-agent schemes now moving through standardisation replace “the user agent says it is X” with a verifiable assertion, which is the first real improvement to bot identity in a decade. Where your platform supports verification natively, use its verified-bot classification rather than writing your own user-agent list, which will be stale within a month.
Then write policy in layers rather than one switch. Allow verified search crawlers everywhere you want indexed. Decide agent access per path: public catalogue and documentation yes, checkout flows, account areas and anything behind authentication no. Handle training according to your content strategy — block it, allow it, or meter it. Metering is now a real option: the HTTP 402 status has been revived as a pay-per-crawl signal, letting an edge respond with a price rather than a refusal, which we covered in the analysis of the pay-per-crawl economy.
Unverified automation gets the strictest tier by default. That is the correct posture and also the one most likely to catch something you did not intend, so stage it: log-only first, review what would have been blocked, then enforce. The same discipline applies here as in switching on a WAF safely — observation before enforcement, always.
Rate, cache and the cost of being crawled
Blocking is not the only lever, and often not the best one. A crawler you want to keep can still be expensive if it fetches the same expensive dynamic page repeatedly, so shape it. Apply a per-operator rate limit sized to a fraction of your origin capacity, not to your total traffic — if one allowed crawler is generating a noticeable share of origin requests, it is a capacity risk regardless of how welcome it is. Our guide to setting up rate limiting covers the counting models; the relevant point here is that per-operator counting beats per-IP counting for crawler traffic, because crawl fleets rotate addresses by design.
Cache aggressively for the paths crawlers hit. A well-cached documentation or catalogue page costs you almost nothing to serve a thousand times, and a crawler served from cache is a rounding error rather than an origin problem. Where you can, serve crawlers a cacheable variant rather than the personalised render — there is no value in composing a session-specific page for a client that has no session, and doing so needlessly fragments your cache. That is a cache key design question as much as a bot question.
Watch the bandwidth line as well as the origin line. Crawler traffic is real traffic on your invoice, and on metered delivery it is charged like any other byte. If a training crawler you have no relationship with is generating a measurable share of your egress, that is not a philosophical debate about the open web; it is a cost you can decline.
Measure it, then review it quarterly
The metric that makes this decidable is the ratio between what an operator fetches and what it sends back. Count requests per operator over a fixed window, count referred human sessions attributable to the same platform, and divide. The published figures for that ratio vary by orders of magnitude between operators, which is precisely why a single blanket policy is wrong: some crawlers are a genuine channel, others are a pure withdrawal.
Instrument before you decide. Segment logs by verified operator, keep the series long enough to see a trend rather than a spike, and record what you changed and when, so the next quarter’s comparison means something — the log discipline in reading a CDN log line applies directly. Alert on new high-volume agents rather than trying to enumerate them in advance.
Then treat the policy as perishable. Operators split their crawlers, rename them, gain and lose respect for conventions, and platform defaults change under you on announced dates. A quarterly review — ratios, top operators, what your platform’s defaults currently say, and whether your robots.txt still matches your edge rules — is enough to stay ahead of it. Fold it into the same cycle as your security configuration audit so it actually happens.