The Web Is Blocking AI Training, Not AI Search: What 506 Top Sites Actually Do

We checked the robots.txt of 506 of the world’s most-visited websites against 14 AI crawlers. The clearest pattern in the data is one we did not expect, and it reframes a debate that is usually treated as binary.

Websites block AI training crawlers roughly twice as often as they block AI retrieval crawlers. The web is not rejecting AI. It is rejecting being trained on, while keeping the door open to being cited.

The finding

OpenAI, Anthropic and Perplexity each run separate crawlers for separate jobs. One gathers training data. Another indexes content so it can be retrieved and cited when someone asks a question. Site owners are treating these very differently.

Two independent operators, almost identical ratios. That consistency is what makes this look like a considered position rather than noise. Perplexity is the outlier at 1.14×, most likely because its two agents are not split along the training/retrieval line as cleanly in most people’s understanding.

Why this matters

The public conversation about AI crawlers tends to be framed as a single question: block them or allow them. The data says site owners are answering a more precise question — what is this crawler for?

Training access gives a model your content with no attribution, no link and no traffic. Retrieval access puts you in the answer with a citation. For a publisher, those are not remotely the same transaction, and the numbers show they are being priced differently.

The practical implication for anyone running a site: if you have blocked AI crawlers with a blanket rule, you have probably also blocked the retrieval crawlers that would have sent you referral traffic. That is almost certainly not what you intended.

Which crawlers get blocked most

Bytespider is the most-blocked crawler on the web. ByteDance’s agent has a public reputation for crawling aggressively, and site owners are clearly responding to that reputation specifically rather than to AI crawling in general.

CCBot at number two is the strategically significant one. Common Crawl feeds most open training datasets, so blocking it affects many models at once. A site blocking CCBot is making a broader statement than a site blocking a single vendor.

Blocking is deliberate, not accidental

We expected most blocking to be collateral damage — old Disallow: / rules under a wildcard, catching AI crawlers that did not exist when the file was written. The data contradicted that.

Of the 104 sites blocking at least one AI crawler, 95 name that crawler explicitly. Only nine block everything through a blanket wildcard. Among the top of the web, this is a policy decision, not an oversight.

Publishers block hardest

Twelve of the fourteen major news publishers in our sample block at least one AI crawler. Sites blocking twelve or more of the fourteen include nytimes.com, cnn.com, bbc.co.uk, telegraph.co.uk, cnbc.com, reddit.com, facebook.com, pinterest.com, imdb.com and sciencedirect.com.

Amazon is worth singling out: the same twelve-crawler block appears across amazon.com, .co.uk, .de, .co.jp, .ca and .in. That is a centrally managed global policy, applied consistently.

Methodology

Domains were drawn from the Tranco top-1M list, downloaded 26 July 2026. Tranco is a peer-reviewed, research-oriented ranking designed to resist manipulation, and has been used in over 350 academic studies.

Infrastructure, CDN and DNS domains were excluded by pattern. For each remaining domain we fetched /robots.txt with a standard browser user-agent, a five-second timeout and up to three redirects, in chunks of forty concurrent requests. Files were parsed into user-agent groups honouring consecutive User-agent lines, and for each crawler the most specific matching group was resolved — exact name first, falling back to *.

A crawler was recorded as blocked when Disallow: / applied with no Allow exceptions, partial when some paths were disallowed, and allowed otherwise. 845 domains were scanned; 506 returned a reachable robots.txt.

Limitations

We hit two methodology problems during this study and corrected both. They are worth stating plainly, because either would have produced a false headline.

• Tranco’s raw top list is not a list of websites. Its highest ranks are dominated by infrastructure domains — DNS roots, CDN endpoints, asset hosts — which serve no robots.txt because they serve no site. An unfiltered run would have reported that most top sites have no robots.txt, which is simply untrue.
• High request concurrency silently corrupts results. At 175 parallel requests, google.com appeared unreachable. At 25 to 40 it fetches normally. Our first pass was built on data degraded this way and was discarded.

Beyond those: robots.txt records stated policy, not enforcement — this measures what sites declare, not whether crawlers comply. The reach rate was 59.9%, with the remainder timing out or refusing connections, some of which is bot protection rather than genuine unavailability. The sample skews toward large, Western sites. And it is a snapshot; robots.txt changes.

The data

The full dataset is published openly — 845 rows, one per domain, with per-crawler verdicts.

Download the dataset (CSV)

If you use it, a link back is appreciated. We intend to re-run this quarterly so the trend becomes visible.

The follow-up study

Access is only half the question. We also scored 243 of these sites on whether an AI can actually extract an answer from them — the median was 45 out of 100.

Check your own site

The same checks used in this study are available as a free tool. It reads your robots.txt, resolves which rules apply to each of the fourteen crawlers, and shows the exact directive responsible for every verdict.

Run the AI Crawler Access Checker

If you find you are blocking retrieval crawlers as well as training crawlers, that is the pattern this study suggests most site owners did not intend.

This is part of an ongoing measurement programme. See all studies, datasets, methodology and our corrections log on the MoxSEO Research page.