Find out which AI systems are allowed to read your site
Your robots.txt decides whether ChatGPT, Claude, Perplexity, Gemini and Apple Intelligence can see your content. Most sites have never checked, and a surprising number are blocking the assistants they most want to appear in.
Your robots.txt was written for Google. AI crawlers are not Google.
Most robots.txt files were set up years ago, by a developer, for one purpose: keeping search engines out of admin pages. Then a dozen AI crawlers appeared, each with its own user-agent, and almost nobody went back to check what those rules now imply.
Blanket rules catch everything
A single User-agent: * with Disallow: / blocks every AI crawler alongside every search engine. Plugins and security suites add these silently, often during a staging-to-live migration that nobody audited afterwards.
Training and retrieval are different
GPTBot trains models. OAI-SearchBot indexes for ChatGPT Search. Blocking the first does not remove you from the second, and blocking both removes you from answers entirely. Very few sites make that distinction deliberately.
Nothing tells you it happened
There is no Search Console for AI visibility. If ClaudeBot has been blocked for eighteen months, no dashboard flags it, no email arrives, and your traffic reports look exactly the same as they always did.
Fourteen crawlers, grouped by who operates them and what they actually do
OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User
GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT Search results. ChatGPT-User fetches a page live when someone pastes your link into a conversation. Blocking all three removes you from ChatGPT completely.
Anthropic — ClaudeBot, Claude-SearchBot, Claude-User
The same three-way split: training, search indexing, and live user-triggered fetches. Claude cites sources prominently in answers, so retrieval access matters more here than training access does.
Perplexity — PerplexityBot, Perplexity-User
Perplexity is answer-first and citation-heavy, which makes it one of the highest-value referral sources in AI search. Blocking PerplexityBot removes you from a channel that actually sends clicks.
Google — Google-Extended
A separate token that controls Gemini and AI Overviews training only. It does not affect Googlebot or your normal search rankings — a distinction many site owners get wrong in both directions.
Apple — Applebot-Extended
Governs whether your content trains Apple Intelligence. Applebot itself still crawls for Siri and Spotlight; the Extended token is training-only.
Others — CCBot, Bytespider, Amazonbot, meta-externalagent
Common Crawl feeds most open training datasets, so blocking CCBot has downstream effects far beyond one model. The rest cover ByteDance, Alexa and Rufus, and Meta AI respectively.
Check your robots.txt against every major AI crawler
Enter any domain. We fetch its public robots.txt, resolve which rules apply to each crawler, and show you the exact directive responsible for every verdict.
Reads only the public robots.txt and llms.txt. No crawling, no signup, nothing stored.
We ran this check across 506 of the world’s top websites
Training crawlers are blocked roughly twice as often as retrieval crawlers. Full findings, methodology and open dataset. Read the study →
What each verdict actually means for your AI visibility
The crawler can read everything
No rule blocks this user-agent. Your content is eligible for training, retrieval and citation, depending on what that particular crawler does.
Some paths are blocked
Usually harmless when it is only /wp-admin/. Worth investigating when the disallowed paths include your blog, docs or product pages — that is where citations come from.
Sitewide Disallow: / applies
This crawler sees nothing. If it was deliberate, fine. If it was inherited from a staging config or a security plugin, you are invisible in that assistant for no reason.
Blocking AI crawlers is a legitimate choice. Doing it by accident is not.
Reasons to allow
AI assistants are becoming a discovery channel. Perplexity and ChatGPT Search cite sources and send referral traffic. If you publish content to be found, blocking the systems people increasingly ask first works against you. Retrieval crawlers in particular tend to reward good structure rather than exploit it.
Reasons to block
If your content is your product — original research, paid courses, licensed material, a proprietary database — training crawlers extract value without attribution or payment. Publishers with licensing deals often block by default and negotiate access separately. That is a sound commercial position.
The point of this tool is not to push you either way. It is to make sure whichever position you are in is the one you chose.
AI crawler access, answered plainly
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended controls Gemini and AI Overviews training only. Googlebot handles Search and is governed separately. You can block one and keep the other, and many publishers do exactly that.
If I block GPTBot, am I out of ChatGPT?
Not entirely. GPTBot is the training crawler. OAI-SearchBot builds the index used for ChatGPT Search, and ChatGPT-User fetches pages when a person shares a link. To be genuinely absent you would need to block all three.
Do AI crawlers actually obey robots.txt?
The major ones from OpenAI, Anthropic, Google, Apple and Perplexity publicly commit to honouring it, and independent testing broadly supports that. Some smaller scrapers ignore it entirely. robots.txt is a request, not a firewall — if you need enforcement, that happens at server or WAF level.
What is llms.txt and do I need one?
robots.txt says what crawlers may not read. llms.txt tells them what is worth reading and how your site is organised. It is a young convention with real adoption momentum. Our llms.txt tool validates yours or builds a starter file from your sitemap.
How often should I re-check this?
After any migration, security plugin change, or CDN configuration update — those are the three things that most often rewrite robots.txt without anyone noticing. Quarterly is sensible otherwise, since new AI user-agents appear regularly.
Knowing you are crawlable is the floor, not the ceiling
Access gets you eligible. Being chosen as the cited source is a different problem — one of structure, entity clarity and demonstrable expertise.