AI crawler access checker
The AI crawler access checker reads your site's robots.txt and tells you, for 14 AI crawlers including GPTBot, ClaudeBot, PerplexityBot and Google-Extended, whether each is allowed, partly blocked or blocked, and which rule decided it. It also checks whether your site publishes an llms.txt file.
- Crawlers
- 14 user agents
- Reads
- public robots.txt only
- Cost
- free, no signup
Fourteen crawlers
Open any crawler for what it does; after a check, the rule matched appears
-
Not runGPTBot (OpenAI)—
Why it mattersOpenAI's training crawler. Content it collects may be used to train OpenAI's foundation models. Blocking it does not remove you from ChatGPT search.
How to fixAllow it if you are happy for public content to inform model training; block it with its own User-agent group if you are not.
-
Not runOAI-SearchBot (OpenAI)—
Why it mattersSurfaces websites in ChatGPT's search features. Blocking it keeps your pages out of ChatGPT search results.
How to fixKeep it allowed if you want to be found and cited in ChatGPT search, even if you block GPTBot.
-
Not runChatGPT-User (OpenAI)—
Why it mattersFetches a page when a ChatGPT user's request needs it, for example after sharing a link. OpenAI notes robots.txt rules may not apply to these user-initiated visits.
How to fixLeave it allowed unless you have a reason to refuse live fetches; enforce at server level if you must.
-
Not runClaudeBot (Anthropic)—
Why it mattersAnthropic's crawler for web content that may contribute to model training.
How to fixDecide on training separately from search: you can block ClaudeBot and still allow Claude-SearchBot.
-
Not runClaude-SearchBot (Anthropic)—
Why it mattersCrawls to improve Claude's search results. Blocking it keeps your pages from being indexed for Claude search.
How to fixKeep it allowed if you want to appear in answers that Claude builds from search.
-
Not runClaude-User (Anthropic)—
Why it mattersFetches pages when someone asks Claude a question that needs them. Blocking it stops your content being retrieved for those requests.
How to fixLeave it allowed if you want Claude to read your pages when users ask about you.
-
Not runPerplexityBot (Perplexity)—
Why it mattersIndexes pages for Perplexity's answers, which cite their sources with links.
How to fixKeep it allowed if you want to be eligible for Perplexity citations.
-
Not runPerplexity-User (Perplexity)—
Why it mattersFetches a page when a Perplexity user's question needs it.
How to fixLeave it allowed unless you want to refuse live, user-triggered fetches.
-
Not runGoogle-Extended (Google)—
Why it mattersA robots.txt control token, not a separate crawler. Google says it controls whether crawled content is used to train future Gemini models and for grounding, and that it does not affect inclusion or ranking in Google Search.
How to fixBlock it if you do not want your content used for Gemini training and grounding. Googlebot and your rankings are unaffected.
-
Not runApplebot-Extended (Apple)—
Why it mattersA control token, not a crawler. It lets you opt out of content crawled by Applebot being used to train Apple's foundation models, which power Apple Intelligence.
How to fixBlock it to opt out of Apple's model training while Applebot keeps crawling for Siri, Spotlight and Safari.
-
Not runCCBot (Common Crawl)—
Why it mattersBuilds the open Common Crawl archive, which many open training datasets draw on, so a block here reaches beyond one company.
How to fixDecide deliberately. Check whether your site is already in recent crawls with the Common Crawl Visibility Checker.
-
Not runBytespider (ByteDance)—
Why it mattersByteDance's crawler.
How to fixAllow or block it with its own User-agent group, in line with your policy on AI training.
-
Not runAmazonbot (Amazon)—
Why it mattersAmazon's crawler, associated with services such as Alexa and Rufus.
How to fixAllow or block it with its own User-agent group.
-
Not runmeta-externalagent (Meta)—
Why it mattersMeta's crawler associated with training Llama and Meta AI.
How to fixAllow or block it with its own User-agent group, in line with your policy on AI training.
What the checker read
robots.txt status, size and user-agent groups, plus llms.txt
Nothing to show yet. After a check, this shows whether robots.txt was found, its size, how many user-agent groups it declares, and whether llms.txt is published.
Allowed, partial, or blocked
For each crawler the checker finds the user-agent group that names it. A crawler with its own group ignores the User-agent: * group entirely; only when it has no group of its own do the * rules apply. The verdict then depends on the Disallow and Allow lines in that one group.
| Verdict | Status shown | When it applies |
|---|---|---|
| Allowed | Pass | No group applies, or the applicable group has no Disallow rule. A missing robots.txt means every crawler is allowed by default. |
| Partial | Needs work | Some paths are disallowed, or the group has Disallow: / with one or more Allow exceptions. |
| Blocked | Fail | Disallow: / applies with no Allow exceptions, under the crawler's own group or, failing that, under *. |
Eligible for whatever that crawler does: training, search or live fetches.
Harmless if only /wp-admin/ is blocked; worth fixing if your blog, docs or product pages are.
Fine if deliberate. If it came from a staging config or plugin, you are invisible to that system for no reason.
The checker reads robots.txt only. It does not see firewall, CDN or bot-management blocking, meta robots tags or X-Robots-Tag headers, so a crawler shown as allowed can still be refused by your server. robots.txt is a request, not enforcement: Google's robots.txt introduction notes it is up to each crawler to obey it.
506top domains checked for AI crawler rules
We ran this check across 506 of the world's top websites. Training crawlers were blocked about twice as often as retrieval crawlers.
- 74domains blocked ClaudeBot, against 39 that blocked Claude-SearchBot.
- 72domains blocked GPTBot, against 37 that blocked OAI-SearchBot.
- 2xTraining crawlers are blocked roughly twice as often as the crawlers that power AI search and citations.
- ClaudeBot (training)74 of 506
- GPTBot (training)72 of 506
- Claude-SearchBot (search)39 of 506
- OAI-SearchBot (search)37 of 506
Counts from the MoxSEO AI crawler-blocking study. Percentages are counts divided by 506.
How to run and act on the check
About five seconds per domain. Run it after any migration, security plugin change or CDN update.
-
Step 1
Enter the domain
Type the domain, such as yourbrand.com. The checker fetches https://yourbrand.com/robots.txt from our server, following up to three redirects.
-
Step 2
Read the headline count
The summary shows how many of the 14 crawlers are allowed and how many are blocked, and says whether robots.txt was found at all.
-
Step 3
Open each blocked or partial row
Every row shows what the crawler does and the rule matched, for example
Disallow: / under User-agent: *, so you know exactly which line to change. -
Step 4
Separate training from search
Decide per crawler. Many sites block training crawlers such as GPTBot and ClaudeBot but keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed so they stay citable.
-
Step 5
Edit robots.txt and re-check
Give each crawler its own User-agent group if it needs different rules from *, publish, and run the check again to confirm the verdicts changed.
Why robots.txt rules catch AI crawlers by accident
Most robots.txt files were written years ago to keep search engines out of admin pages. A single User-agent: * group with Disallow: /, often left over from a staging site or added by a security plugin, blocks every AI crawler along with every search engine. Nothing reports it: there is no Search Console for AI visibility, so a block can sit unnoticed for months.
The other common mistake is treating all AI crawlers as one. OpenAI documents that each of its crawlers is controlled independently: GPTBot collects training data, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches pages on a user's behalf. Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User the same way. Google-Extended and Applebot-Extended are not crawlers at all but tokens that control how content their main crawlers already fetched is used, as Google and Apple document.
Blocking is a legitimate choice. If your content is your product, such as original research, paid courses or licensed data, blocking training crawlers is a sound commercial position. If you publish to be found, blocking the systems people now ask first works against you. The point of this checker is to make sure the position you are in is the one you chose. Once access is right, check the llms.txt file and whether pages are structured to be quoted.
| Job | Crawlers and tokens |
|---|---|
| Model training | GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider |
| Training and grounding controls | Google-Extended, Applebot-Extended |
| AI search indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Live fetch on a user's request | ChatGPT-User, Claude-User, Perplexity-User |
| Assistant and shopping services | Amazonbot |
What the checker does
- Fetches the public robots.txt of one domain
- Resolves the applicable user-agent group for 14 AI crawlers
- Shows the rule matched for every verdict
- Reports whether /llms.txt is published
What it does not do
- See firewall, CDN or bot-management blocks
- Read meta robots tags or X-Robots-Tag headers
- Check whether a crawler actually visited you (use your server logs)
- Crawl any page other than robots.txt and llms.txt
Need help reviewing your crawler rules?
Review training and retrieval crawler rules separately, with your access policy and website setup in view.
AI crawler access, answered plainly
Short, standalone answers. Email editorial@moxseo.com if any of them is wrong.
What does the AI Crawler Access Checker do?
It fetches a domain's public robots.txt and works out, for 14 AI crawlers and control tokens, whether each is allowed, partly blocked or blocked, showing the rule that decided it. It also checks whether the domain publishes an llms.txt file. It does not crawl any other page.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether content Google crawls is used to train future Gemini models and for grounding. Googlebot, which handles Search, is governed separately.
If I block GPTBot, am I out of ChatGPT?
No. GPTBot is OpenAI's training crawler. OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches pages when a user asks. OpenAI says each setting is independent, so blocking GPTBot only signals that your content should not be used for training.
Do AI crawlers obey robots.txt?
The major operators say their crawlers respect it, but robots.txt is a request, not enforcement, and some scrapers ignore it. OpenAI also notes that robots.txt rules may not apply to user-initiated fetches by ChatGPT-User. If you need a hard block, enforce it at your server, CDN or firewall.
Why does a crawler show as allowed when I added rules for all bots?
Because a crawler with its own User-agent group ignores the User-agent: * group completely. If robots.txt has a group for GPTBot with no Disallow line, GPTBot is allowed even when * is blocked. The checker shows the rule it matched for every crawler.
What does a missing robots.txt mean?
If robots.txt does not exist or does not return a plain-text file, every crawler is allowed by default. The checker reports the HTTP status it received and marks all 14 crawlers as allowed.
What is llms.txt and do I need one?
robots.txt says what crawlers may not read; llms.txt is a proposed Markdown file that points language models to the pages worth reading. It is a young convention, not a ranking factor. The llms.txt Generator and Validator checks an existing file or drafts a starter from your sitemap.
How often should I re-check?
After any site migration, security plugin change or CDN configuration update, since those most often rewrite robots.txt without anyone noticing. Otherwise every quarter, because new AI user agents appear regularly.
Run the next check
All 18 free tools- llms.txt Generator and ValidatorValidate your llms.txt or draft one from the sitemap.
- AI Fetch Timeout Risk CheckerAn allowed crawler still needs the page to load in time.
- Common Crawl Visibility CheckerSee whether CCBot has captured your domain in recent crawls.
- AI Answer Extractability CheckerOnce crawlers can read the page, can they quote it?
- Agentic Resource Discovery CheckerCheck whether AI agents can find a machine-readable catalogue.
Crawlable is the floor,
not the ceiling.
Access makes you eligible; being chosen as the cited source is a question of structure, entity clarity and expertise. Book a free 30-minute call with Sakshi Kumari or Ashish Khan, or see our AI SEO services.
A call is not an engagement. We never guarantee rankings or AI citations.