Last reviewed 19 October 2026.
We checked the robots.txt of 506 of the world’s most-visited websites against 14 AI crawlers. The clearest pattern in the data is one we did not expect, and it reframes a debate that is usually treated as binary.
Websites block AI training crawlers roughly twice as often as they block AI retrieval crawlers. The web is not rejecting AI. It is rejecting being trained on, while keeping the door open to being cited.
Do websites block AI training and AI search crawlers differently?
Yes. OpenAI, Anthropic and Perplexity each run separate crawlers for separate jobs. One gathers training data. Another indexes content so it can be retrieved and cited when someone asks a question. Site owners are treating these very differently.
Fig. 1Sites blocking each crawler, by operator, n = 506
| Operator | Training crawler | Blocked | Retrieval crawler | Blocked | Ratio |
|---|---|---|---|---|---|
| OpenAI | GPTBot | 72 | OAI-SearchBot | 37 | 1.95× |
| Anthropic | ClaudeBot | 74 | Claude-SearchBot | 39 | 1.90× |
| Perplexity | PerplexityBot | 57 | Perplexity-User | 50 | 1.14× |
Two independent operators, almost identical ratios. That consistency is what makes this look like a considered position rather than noise. Perplexity is the outlier at 1.14×, most likely because its two agents are not split along the training/retrieval line as cleanly in most people’s understanding.
Why does the training and retrieval split matter?
The public conversation about AI crawlers tends to be framed as a single question: block them or allow them. The data says site owners are answering a more precise question: what is this crawler for?
Training access gives a model your content with no attribution, no link and no traffic. Retrieval access puts you in the answer with a citation. For a publisher, those are not remotely the same transaction, and the numbers show they are being priced differently.
The web is not rejecting AI. It is rejecting being trained on, while keeping the door open to being cited.
The practical implication for anyone running a site: if you have blocked AI crawlers with a blanket rule, you have probably also blocked the retrieval crawlers that would have sent you referral traffic. That is almost certainly not what you intended. OpenAI, for example, documents its crawlers separately, so each one can be allowed or refused by name.
Which AI crawlers get blocked most?
Fig. 2All 14 crawlers ranked by sites blocking, n = 506
| # | Crawler | Operator | Job in Fig. 1 | Sites blocking |
|---|---|---|---|---|
| 1 | Bytespider | ByteDance | Not classified | 85 |
| 2 | CCBot | Common Crawl | Not classified | 81 |
| 3 | ClaudeBot | Anthropic | Training | 74 |
| 4 | GPTBot | OpenAI | Training | 72 |
| 5 | meta-externalagent | Meta | Not classified | 67 |
| 6 | Google-Extended | Not classified | 62 | |
| 7 | Applebot-Extended | Apple | Not classified | 60 |
| 8 | PerplexityBot | Perplexity | Training | 57 |
| 9 | Amazonbot | Amazon | Not classified | 52 |
| 10 | Perplexity-User | Perplexity | Retrieval | 50 |
| 11 | ChatGPT-User | OpenAI | Not classified | 47 |
| 12 | Claude-User | Anthropic | Not classified | 46 |
| 13 | Claude-SearchBot | Anthropic | Retrieval | 39 |
| 14 | OAI-SearchBot | OpenAI | Retrieval | 37 |
Bytespider is the most-blocked crawler on the web. ByteDance’s agent has a public reputation for crawling aggressively, and site owners are clearly responding to that reputation specifically rather than to AI crawling in general.
CCBot at number two is the strategically significant one. Common Crawl feeds most open training datasets, so blocking it affects many models at once. A site blocking CCBot is making a broader statement than a site blocking a single vendor.
Is AI crawler blocking deliberate or accidental?
Deliberate. We expected most blocking to be collateral damage: old Disallow: / rules under a wildcard, catching AI crawlers that did not exist when the file was written. The data contradicted that.
Fig. 3Share of the 506 sites blocking at least one AI crawler
20.6%
| Metric | Sites | % |
|---|---|---|
| Serving a real robots.txt | 377 | 74.5% |
| Naming at least one AI crawler explicitly | 143 | 28.3% |
| Blocking at least one AI crawler | 104 | 20.6% |
| Blocking via a named rule (deliberate) | 95 | 18.8% |
| Blocking all 14 crawlers | 13 | 2.6% |
| Blocking all 14 via wildcard only | 9 | 1.8% |
Of the 104 sites blocking at least one AI crawler, 95 name that crawler explicitly. Only nine block everything through a blanket wildcard. Among the top of the web, this is a policy decision, not an oversight.
Who blocks AI crawlers hardest?
Publishers. Twelve of the fourteen major news publishers in our sample block at least one AI crawler. Sites blocking twelve or more of the fourteen include nytimes.com, cnn.com, bbc.co.uk, telegraph.co.uk, cnbc.com, reddit.com, facebook.com, pinterest.com, imdb.com and sciencedirect.com.
Amazon is worth singling out: the same twelve-crawler block appears across amazon.com, .co.uk, .de, .co.jp, .ca and .in. That is a centrally managed global policy, applied consistently.
Methodology
Domains were drawn from the Tranco top-1M list, a peer-reviewed, research-oriented ranking designed to resist manipulation, used in over 350 academic studies.
- Sample
- Tranco top-1M, downloaded 26 July 2026. Infrastructure, CDN and DNS domains excluded by pattern.
- Reach
- 845 domains scanned; 506 returned a reachable robots.txt (59.9%).
- Fetch
/robots.txtwith a standard browser user-agent, a five-second timeout and up to three redirects, in chunks of forty concurrent requests.- Parsing
- User-agent groups honouring consecutive
User-agentlines. For each crawler the most specific group wins: exact name first, falling back to*. - Verdict
- Blocked when
Disallow: /applied with noAllowexceptions; partial when some paths were disallowed; allowed otherwise.
Limitations
Found during the study and corrected
- Raw Tranco is not a list of websites. Its highest ranks are DNS roots, CDN endpoints and asset hosts that serve no robots.txt. Unfiltered, the study would have reported that most top sites have no robots.txt, which is untrue. Corrected.
- High concurrency silently corrupts results. At 175 parallel requests google.com appeared unreachable; at 25 to 40 it fetches normally. The first pass was built on degraded data and discarded. Corrected.
- robots.txt records stated policy, not enforcement. This measures what sites declare, not whether crawlers comply.
- The reach rate was 59.9%. The rest timed out or refused connections, some of it bot protection rather than genuine unavailability.
- The sample skews toward large, Western sites, and it is a snapshot: robots.txt changes.
Can I download the data?
Yes. The full dataset is published openly: 845 rows, one per domain, with per-crawler verdicts.
ai-crawler-study-2026-07-26.csv845 rows, per-crawler verdictsIf you use it, a link back is appreciated. We intend to re-run this quarterly so the trend becomes visible.
The follow-up study
Access is only half the question. We also scored 243 of these sites on whether an AI can actually extract an answer from them: the median was 45 out of 100, and only 3% were structurally quotable.
How do I check my own robots.txt?
The same checks used in this study are available as a free tool. It reads your robots.txt, resolves which rules apply to each of the fourteen crawlers, and shows the exact directive responsible for every verdict. Run the AI Crawler Access Checker, or follow Ashish’s step-by-step guide to auditing robots.txt for AI crawler access.
If you find you are blocking retrieval crawlers as well as training crawlers, that is the pattern this study suggests most site owners did not intend. If AI visibility is a priority, our AI SEO service starts with exactly this audit.
This is part of an ongoing measurement programme. See all studies, datasets and the methodology on the MoxSEO Research page.
Questions about this study
Which AI crawlers do websites block most?
Bytespider, run by ByteDance, is the most-blocked AI crawler in our scan of 506 top websites: 85 sites block it. CCBot (Common Crawl) follows at 81, then ClaudeBot at 74 and GPTBot at 72. The least-blocked are the two search crawlers that cite sources in answers, Claude-SearchBot (39) and OAI-SearchBot (37).
What is the difference between an AI training crawler and a retrieval crawler?
A training crawler, such as GPTBot or ClaudeBot, collects pages to train a model, with no link or attribution back to you. A retrieval crawler, such as OAI-SearchBot or Claude-SearchBot, fetches pages so an AI answer can quote and cite them. Blocking the first keeps your content out of training data; blocking the second keeps you out of cited answers.
How many top websites block AI crawlers?
104 of the 506 sites with a reachable robots.txt, or 20.6%, block at least one of the 14 AI crawlers we checked. 95 of those 104 name the crawler explicitly, only nine block through a wildcard rule alone, and 13 sites block all 14 crawlers.
Does blocking GPTBot keep my site out of ChatGPT search?
Not on its own. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler that surfaces sites in ChatGPT search (platform.openai.com/docs/bots). A rule that names only GPTBot leaves OAI-SearchBot allowed. A blanket "User-agent: *" with "Disallow: /" blocks both.
Is a robots.txt block enforced?
No. robots.txt is a stated policy under the Robots Exclusion Protocol (RFC 9309), not an access control. This study measures what sites declare, not whether each crawler complies.
Can I use the dataset?
Yes. The full CSV is published openly: 845 rows, one per domain scanned, with a verdict for each of the 14 crawlers. A link back is appreciated. We intend to re-run the scan quarterly so the trend becomes visible.
How do I check which AI crawlers my own robots.txt blocks?
Run your domain through the free AI Crawler Access Checker. It reads your robots.txt, resolves which rule applies to each of the 14 crawlers in this study, and shows the exact directive responsible for every verdict.