Common Crawl visibility checker
The Common Crawl visibility checker tells you whether your domain appears in the three most recent Common Crawl indexes. For each crawl it asks Common Crawl's own index whether any page on your domain was captured with a 200 status, and reports captured, not captured or index unavailable. It does not count pages.
- Crawls checked
- the 3 most recent
- Source
- Common Crawl's public index
- Cost
- free, no signup
Three recent crawls
After a check, each row is one Common Crawl index and whether it captured your domain
-
Not runCaptured—
Why it mattersThe crawl's index holds at least one capture of a page on your domain that returned HTTP 200. Common Crawl reached the site during that crawl.
How to fixNothing to fix. Captures are a record of discovery at that point in time, not a guarantee every important page was fetched.
-
Not runNot captured—
Why it mattersThe index answered but holds no status-200 capture for the domain. Crawl selection, robots.txt rules, blocking at a firewall or CDN, redirects to another host, or a new site can all explain it.
How to fixCheck robots.txt rules for CCBot with the AI Crawler Access Checker, confirm the canonical host answers 200 without a challenge page, and make sure important pages are linked and in your sitemap.
-
Not runIndex unavailable—
Why it mattersCommon Crawl's index server did not answer for that crawl, so the checker cannot say either way.
How to fixRun the check again later. An unavailable index says nothing about your site.
-
Not runWhich crawls are checked—
Why it mattersThe checker reads Common Crawl's published index list and takes the three most recent crawls. Crawl IDs follow the pattern CC-MAIN-year-week.
How to fixLook for a pattern across the three rows: captured in every crawl, intermittently, or not at all.
Note from the lookup
What the result can and cannot tell you
Nothing to show yet. After a check, a short note on how to read the result appears here.
What is Common Crawl, and why check it?
Common Crawl is a non-profit that maintains a free, open archive of web crawl data, collected by its crawler, CCBot. The archive is used by researchers, search tools and dataset builders, including some AI-development workflows.
Checking it answers a narrow, useful question: did CCBot reach your domain in recent crawls? If it did not, the cause is often the same thing that stops other crawlers, such as a robots.txt rule, a challenge page at the CDN or a redirect to a different host.
It is an observation, not a score. A capture shows the domain was discovered at that point in time. It cannot prove that every important page was captured, that any AI system trained on the content, or that the site will appear in an answer.
For background, read whether your content is in Common Crawl and why it matters for AI.
| Step | What happens |
|---|---|
| 1. Index list | Reads Common Crawl's published list of crawl indexes (collinfo.json) and takes the three most recent. |
| 2. Domain query | Asks each crawl's CDX index for captures matching your domain, across all paths. |
| 3. Status filter | Counts only captures that returned HTTP 200, so redirects and errors do not show as captured. |
| 4. Result | Reports captured, not captured or index unavailable for each crawl, and how many of the three captured the domain. |
How to use this result
-
Step 1
Enter the canonical domain
Use the hostname people and crawlers should reach. If www and non-www, or http and https, redirect inconsistently, a crawler may capture the redirect rather than the page.
-
Step 2
Read the three rows as observations
Captured means CCBot fetched a page that returned 200 during that crawl. Missing records may reflect crawl selection, access restrictions, a new site or incomplete coverage.
-
Step 3
If captures are missing, inspect access
Check robots.txt for CCBot rules with the AI Crawler Access Checker, then response status, canonical redirects, crawlable navigation, sitemaps and internal links to important pages.
-
Step 4
Decide whether you want to be included
Blocking CCBot is a legitimate choice if you do not want your content in open datasets. The point is that it should be a decision, not an accident.
-
Step 5
Re-check after the next crawl
Changes only show once a new crawl is indexed, so check again when a newer crawl ID appears.
What a missing record does not prove
Do not use a missing Common Crawl record as proof that a site is invisible to Google, ChatGPT or another platform. Each system has its own crawlers, indexes, permissions and retrieval behaviour. In our AI crawler-blocking study of 506 domains, sites set different rules for different bots, so one crawler's access tells you little about another's.
A record also does not show whether a live answer engine can fetch the page quickly today. Test that with the AI Fetch Timeout Risk Checker.
What the checker does
- Queries the three most recent Common Crawl indexes
- Matches the whole domain, including www and other subdomains, all paths
- Counts only status-200 captures
- Flags any index that did not answer
What it does not do
- Count how many of your URLs are indexed
- Show which pages were captured
- Tell you whether any model trained on your content
- Check Google, Bing or any AI index
Need help interpreting crawl history?
Discuss what the returned crawl records show, what they do not establish and which checks to run next.
Common Crawl, answered plainly
Short, standalone answers. Email editorial@moxseo.com if any of them is wrong.
How do I check if my site is in Common Crawl?
Enter your domain in the Common Crawl Visibility Checker. It reads Common Crawl's list of crawl indexes, queries the three most recent ones for any page on your domain that returned HTTP 200, and shows captured, not captured or index unavailable for each crawl.
Does a capture mean my content is in AI training data?
No. A capture shows Common Crawl fetched a page from your domain during that crawl. Whether any particular model was trained on that crawl, or on your pages, is not something the index records.
Why does my domain show as not captured?
Common Crawl does not fetch every site in every crawl. A robots.txt rule for CCBot, a firewall or CDN challenge, redirects to another host, very few inbound links or a new domain can all mean no status-200 capture was recorded.
Does the checker tell me how many pages are indexed?
No. It checks whether at least one status-200 capture exists for the domain in each crawl. It does not estimate the number of URLs captured or list which pages they were.
What does index unavailable mean?
Common Crawl's index server did not answer for that crawl when the checker asked. It says nothing about your site. Run the check again later.
How do I stop Common Crawl from crawling my site?
Common Crawl's crawler identifies itself as CCBot. A robots.txt group for user-agent CCBot with Disallow: / asks it not to crawl the site. Check what your robots.txt currently says with the AI Crawler Access Checker.
Is a missing record proof that Google or ChatGPT cannot see my site?
No. Google, OpenAI and other platforms run their own crawlers and indexes with their own rules. Common Crawl is one public archive, so a gap there is a prompt to check access, not a verdict.
Run the next check
All 24 free tools- AI Crawler Access CheckerSee whether robots.txt allows CCBot, GPTBot, ClaudeBot and 11 more.
- AI Fetch Timeout Risk CheckerCheck whether a live AI fetch can retrieve the page in time.
- llms.txt Generator and ValidatorPoint language models at the pages you want read first.
- On-Page SEO AuditStatus, redirects, canonical and noindex for any URL.
Want a clear view of who can read
your site?
A free 30-minute call with Sakshi Kumari or Ashish Khan. Bring this result; they will check crawler access, rendering and indexability with you and name what needs implementation work.
A call is not an engagement. We never guarantee rankings or AI citations.