Your robots.txt decides whether ChatGPT, Claude, Perplexity, Gemini and Apple Intelligence can see your content. Most sites have never checked, and a surprising number are blocking the assistants they most want to appear in.
Most robots.txt files were set up years ago, by a developer, for one purpose: keeping search engines out of admin pages. Then a dozen AI crawlers appeared, each with its own user-agent, and almost nobody went back to check what those rules now imply.
A single User-agent: * with Disallow: / blocks every AI crawler alongside every search engine. Plugins and security suites add these silently, often during a staging-to-live migration that nobody audited afterwards.
GPTBot trains models. OAI-SearchBot indexes for ChatGPT Search. Blocking the first does not remove you from the second, and blocking both removes you from answers entirely. Very few sites make that distinction deliberately.
There is no Search Console for AI visibility. If ClaudeBot has been blocked for eighteen months, no dashboard flags it, no email arrives, and your traffic reports look exactly the same as they always did.
GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT Search results. ChatGPT-User fetches a page live when someone pastes your link into a conversation. Blocking all three removes you from ChatGPT completely.
The same three-way split: training, search indexing, and live user-triggered fetches. Claude cites sources prominently in answers, so retrieval access matters more here than training access does.
Perplexity is answer-first and citation-heavy, which makes it one of the highest-value referral sources in AI search. Blocking PerplexityBot removes you from a channel that actually sends clicks.
A separate token that controls Gemini and AI Overviews training only. It does not affect Googlebot or your normal search rankings — a distinction many site owners get wrong in both directions.
Governs whether your content trains Apple Intelligence. Applebot itself still crawls for Siri and Spotlight; the Extended token is training-only.
Common Crawl feeds most open training datasets, so blocking CCBot has downstream effects far beyond one model. The rest cover ByteDance, Alexa and Rufus, and Meta AI respectively.
Enter any domain. We fetch its public robots.txt, resolve which rules apply to each crawler, and show you the exact directive responsible for every verdict.
Reads only the public robots.txt and llms.txt. No crawling, no signup, nothing stored.
Training crawlers are blocked roughly twice as often as retrieval crawlers. Full findings, methodology and open dataset. Read the study →
No rule blocks this user-agent. Your content is eligible for training, retrieval and citation, depending on what that particular crawler does.
Usually harmless when it is only /wp-admin/. Worth investigating when the disallowed paths include your blog, docs or product pages — that is where citations come from.
This crawler sees nothing. If it was deliberate, fine. If it was inherited from a staging config or a security plugin, you are invisible in that assistant for no reason.
AI assistants are becoming a discovery channel. Perplexity and ChatGPT Search cite sources and send referral traffic. If you publish content to be found, blocking the systems people increasingly ask first works against you. Retrieval crawlers in particular tend to reward good structure rather than exploit it.
If your content is your product — original research, paid courses, licensed material, a proprietary database — training crawlers extract value without attribution or payment. Publishers with licensing deals often block by default and negotiate access separately. That is a sound commercial position.
The point of this tool is not to push you either way. It is to make sure whichever position you are in is the one you chose.
No. Google-Extended controls Gemini and AI Overviews training only. Googlebot handles Search and is governed separately. You can block one and keep the other, and many publishers do exactly that.
Not entirely. GPTBot is the training crawler. OAI-SearchBot builds the index used for ChatGPT Search, and ChatGPT-User fetches pages when a person shares a link. To be genuinely absent you would need to block all three.
The major ones from OpenAI, Anthropic, Google, Apple and Perplexity publicly commit to honouring it, and independent testing broadly supports that. Some smaller scrapers ignore it entirely. robots.txt is a request, not a firewall — if you need enforcement, that happens at server or WAF level.
robots.txt says what crawlers may not read. llms.txt tells them what is worth reading and how your site is organised. It is a young convention with real adoption momentum. Our llms.txt tool validates yours or builds a starter file from your sitemap.
After any migration, security plugin change, or CDN configuration update — those are the three things that most often rewrite robots.txt without anyone noticing. Quarterly is sensible otherwise, since new AI user-agents appear regularly.
Access gets you eligible. Being chosen as the cited source is a different problem — one of structure, entity clarity and demonstrable expertise.