Robots.txt tester
This robots.txt tester tells you whether a crawler is allowed to fetch a specific URL, and which line of the file decides it. It reads the live robots.txt from the URL's site, or rules you paste before publishing them, and applies the matching rules Google documents. It also compares eight search and AI crawlers on the same path.
- Matching
- RFC 9309, as Google applies it
- Compares
- 8 search and AI crawlers
- Cost
- free, no signup
What the tester found
Before a run, these are the checks made. After a run, each row is a real result, followed by the same path for other crawlers
-
Not runAllowed or blocked—
Why it mattersThis is the answer: whether the crawler you named may fetch this exact path and query string. After a run it shows the deciding rule and its line number.
How to fixTo unblock a URL, remove or narrow the Disallow rule that matched, or add an Allow rule for the path that is longer than it. To block one, add a Disallow rule longer than any Allow that matches. The longest matching rule wins and, on a tie, Allow wins.
-
Not runWhere the rules came from—
Why it mattersRules are read from robots.txt at the root of the URL's protocol, host and port, so https://example.com and https://shop.example.com each need their own file. A 4xx means no restrictions; a 5xx means Google treats the whole site as disallowed while the errors continue.
How to fixA 5xx on robots.txt is urgent: fix the server so the file returns 200, or a 404 if you want no rules.
-
Not runGroup that applies—
Why it mattersA crawler obeys only one set of rules: the groups whose User-agent line names its token, merged together, or the * groups if none do. A crawler with its own group ignores the * group entirely, so a Disallow under * does not apply to it.
How to fixIf the wrong group applies, check the spelling of the User-agent line. If you add a group for a named crawler, copy into it every * rule you still want that crawler to follow.
-
Not runFile size—
Why it mattersGoogle's documentation says it reads the first 500 KiB of a robots.txt file and ignores anything after that. A rule past the limit has no effect on Google.
How to fixShorten the file. Replace long lists of single URLs with directory rules or * patterns.
-
Not runLines crawlers will ignore—
Why it mattersThis row appears only when the file has lines with no colon or with a field the tester does not recognise, such as a misspelt Disalow. Crawlers skip them silently, so a typo can leave a section open.
How to fixCorrect the spelling to User-agent, Allow, Disallow or Sitemap, each followed by a colon. Note that a Noindex line in robots.txt is not supported by Google; use a noindex meta tag or header on a crawlable page instead.
Other crawlers and sitemaps
The same path tested for each crawler in the comparison set, and any Sitemap lines in the file
Nothing to show yet. After a test, this shows a table of Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot on the same path, plus every Sitemap: line in the file.
What each result means
The tester does not score the file. It gives one verdict, allowed or blocked, for one URL and one crawler, and shows the rule that decided it.
| Situation | Verdict | Why |
|---|---|---|
| robots.txt returns 2xx | Rules apply | The file is parsed and the matching group's rules decide. |
| robots.txt returns 4xx | Allowed | No file, so no restrictions. This includes 404 and 410. |
| robots.txt returns 5xx | Blocked | Google treats the whole site as disallowed while the errors continue. |
| robots.txt redirects | Followed | Up to five redirects, then the final response is used. |
| No rule matches the path | Allowed | Crawling is allowed unless a rule says otherwise. |
| An Allow and a Disallow both match | Longest wins | The rule with the most characters wins; on a tie, Allow wins. |
| Empty Disallow line | Matches nothing | It blocks nothing. |
No rule matched, or the longest matching rule is an Allow.
The longest matching rule is a Disallow, or robots.txt returned a server error.
Matching follows RFC 9309 and Google's guide to how Google interprets robots.txt: * matches any run of characters, $ anchors the end of the URL, paths are case-sensitive, and field names and user-agent tokens are not. Like Google, the tester treats a 4xx robots.txt as not found (everything allowed), except 429, which it treats as a server error.
How to test a URL against robots.txt
Test the URLs that matter, not just the homepage.
-
Step 1
Enter the exact URL
Paste the full URL, including the path and any query string. Rules match the path and query together, so
/shop?sort=pricecan be blocked while/shopis allowed. -
Step 2
Name the crawler
Leave the field empty for Googlebot, or type a token such as Bingbot, GPTBot or ClaudeBot. The tester matches the token exactly against your User-agent lines.
-
Step 3
Test live rules or pasted rules
Leave the textarea empty to fetch the live file. To try a change before you publish it, paste the full new file and run the same URL again.
-
Step 4
Read the deciding rule
The verdict row shows the rule and line number that decided the result. If it is not the one you expected, look for a longer matching rule or a crawler-specific group.
-
Step 5
Compare the other crawlers
Check the table of other crawlers for surprises, such as a search crawler caught by an AI-crawler rule.
How crawlers read robots.txt
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler looks for the groups that name its own token and follows only those, merged together. If no group names it, it follows the * groups. It never combines its own group with *, a frequent reason a rule seems to be ignored.
Within the chosen group, the longest matching rule wins, measured in characters, and Allow wins a tie. * matches any run of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDFs without touching /pdf-guide/. Paths are case-sensitive: /Private/ does not cover /private/.
Google retired the robots.txt Tester in Search Console in 2023 and replaced it with the robots.txt report, which shows the files Google found for your property, when it last fetched them and any problems it had reading them. It does not test any URL for any crawler, or rules you have not published yet. That is what this tester is for.
What the tester does
- Fetches robots.txt from the URL's root, following up to five redirects
- Tests pasted rules without publishing or saving them
- Picks the group for the crawler you name, else *
- Applies longest match, Allow on ties, and * and $ wildcards
- Shows the same path for eight search and AI crawlers
What it does not do
- Tell you whether a URL is indexed or ranking
- Read noindex tags, X-Robots-Tag headers or canonicals
- Normalise percent-encoding: the path is matched as you typed it
- Show the cached copy Google actually holds
Crawling is not indexing, and AI crawlers are not one thing
robots.txt controls crawling, not indexing. Google's documentation says a URL blocked by robots.txt can still be indexed if other pages link to it. To keep a page out of results, let it be crawled and add a noindex meta tag or X-Robots-Tag header. If you block the page in robots.txt as well, Google cannot fetch it to see the noindex.
AI crawlers each use their own token, and the owners document different jobs for them. OpenAI lists GPTBot and OAI-SearchBot separately, so you can block one and allow the other. Google-Extended is a product token, not a separate crawler: Google's documentation says it controls whether content is used for its Gemini models, and that it does not affect inclusion in Google Search. Blocking Googlebot to keep content out of AI is a costly mistake.
Each Sitemap: line must be a full URL and applies to every crawler, whatever group it sits next to. Check the files themselves with the XML Sitemap Validator.
Not sure which rules your site actually needs?
We review robots.txt, noindex and sitemaps together, so blocking crawlers never removes pages you want found.
robots.txt, answered plainly
Short, standalone answers. Email editorial@moxseo.com if any of them is wrong.
What does the Robots.txt Tester check?
It takes a URL and a crawler name, fetches robots.txt from the URL's site or uses rules you paste, finds the group that applies to that crawler and decides whether the URL is allowed or blocked, using the longest-match rule Google documents. It shows the deciding rule and its line number, and the result for eight other search and AI crawlers.
Does Google still have a robots.txt tester?
Google retired the robots.txt Tester in Search Console in 2023. It now offers a robots.txt report, which shows the robots.txt files it found for your property, when they were last fetched and any problems reading them. It does not test arbitrary URLs for any crawler, which is what this tester does.
If a URL is blocked in robots.txt, will it disappear from Google?
Not necessarily. robots.txt controls crawling, not indexing. Google's documentation says a blocked URL can still be indexed if other pages link to it. To remove a page from results, allow crawling and add a noindex meta tag or X-Robots-Tag header, so Google can fetch the page and see it.
Why is my Disallow under User-agent: * not blocking Googlebot?
Because your file probably has a separate group for Googlebot. A crawler follows only the groups that name its token and ignores the * group when one exists. Copy the * rules you still want into the Googlebot group, or remove the Googlebot group. The tester's group row shows which group applied.
What happens if robots.txt returns a 404 or a 500?
A 4xx response (other than 429) means there is no robots.txt, so crawlers treat everything as allowed. A 5xx response is different: Google's documentation says it treats the whole site as disallowed while the server errors continue. The tester reports a 5xx as blocked for every crawler, so fix server errors on robots.txt quickly.
Does blocking Google-Extended or GPTBot affect my rankings?
Google's documentation says Google-Extended does not affect inclusion in Google Search; it controls use of content for Google's Gemini models. GPTBot is OpenAI's crawler and does not crawl for Google. Blocking Googlebot itself does affect Search.
Is the tester free, and are my pasted rules stored?
It is free with no signup. Pasted rules are sent to our server, tested and not saved, and they are left out of share links. Live files are fetched from our server, which keeps only a short-lived request counter for rate limiting.
Run the next check
All 23 free tools- AI Crawler Access CheckerSee which AI crawlers your live robots.txt lets in, bot by bot.
- XML Sitemap ValidatorCheck the sitemaps your robots.txt declares for format and broken URLs.
- On-Page and Indexability AuditCheck noindex, canonical and status on a page you can crawl.
- Redirect CheckerTrace the redirect chain behind a URL before you test its rules.
Crawl rules decide what gets fetched.
Everything else decides what ranks.
A clean robots.txt only removes an obstacle; it does not earn rankings on its own. Book a free 30-minute call with Sakshi Kumari or Ashish Khan, or see our SEO services.
A call is not an engagement. We never guarantee rankings or AI citations.