Free AI check Book a strategy call

Research on SEO and AI search Three studies. Three CSVs. One public corrections log.

MoxSEO’s research measures how websites look to AI systems: which crawlers they let in, whether a page offers an answer an AI can lift and credit, and whether a business signs its work. Every study publishes its sample, method, limitations and raw rows, so you can check the finding before you act on it.

Published studiesJul 2026
The three MoxSEO studies, their samples, authors and datasets
StudySampleAuthorData
AI crawler access26 Jul 2026506top sitesAshishKhanCSV
AI answer extractability26 Jul 2026243homepagesSakshiKumariCSV
City-market AI readiness27 Jul 2026148in 20 citiesAshishKhanCSV
One row per site in every file.Corrections log
robots.txt files read, from 845 Tranco top-list domains
506
top-site homepages scored on nine weighted structural checks
243
businesses sampled across 20 cities in seven countries, via Google Places
148
AI crawlers checked in every robots.txt we parsed
14

Three studies, read in order

Each study describes a defined sample and collection window. Study 02 follows up Study 01, which drew its sample from the Tranco top list of 26 July 2026. Study 03 carries the same checks to local businesses in 20 cities. Use the findings to ask better questions and test your own site, not as universal rules for Google or AI platforms.

Study 01Published 26 Jul 2026

The web is blocking AI training, not AI search

Websites block AI training crawlers roughly twice as often as they block the retrieval crawlers that fetch pages to cite in answers. 104 of 506 sites block at least one AI crawler.

Ashish KhanSEO Specialist, MoxSEO

Sample
845 domains from the Tranco top-1M list, infrastructure and CDN domains excluded; 506 returned a reachable robots.txt (59.9%).
Instrument
Each /robots.txt fetched with a standard browser user-agent, 5-second timeout, up to three redirects.
Blocked
Disallow: / applies to the crawler with no Allow exceptions. Some paths disallowed counts as partial.

Stated limitationsrobots.txt records declared policy, not enforcement. Some unreachable sites may sit behind bot protection, and the sample skews toward large Western sites.

1.95×GPTBot is blocked 1.95 times as often as OpenAI's own search crawler.

OpenAI

72GPTBot

37OAI-SearchBot

1.95×
Anthropic

74ClaudeBot

39Claude-SearchBot

1.90×
Perplexity

57PerplexityBot

50Perplexity-User

1.14×
Sites blocking each crawler, n = 506Source: Study 01 dataset

Study 02Published 26 Jul 2026

Only 3% of top websites are structurally quotable by AI

Of the sites AI can reach, how many give it a passage it can lift and credit? We scored 243 homepages. The median was 45 out of 100, and only 5.8% declare an author and dates.

Sakshi KumariSEO Specialist, MoxSEO

Sample
506 Tranco domains attempted; 243 returned a fetchable homepage and were scored (48% yield).
Instrument
Navigation, header, footer, script and form markup stripped, then nine weighted checks: 110 raw points, normalised to 100.
Bands
Strong 80 and above, workable 55 to 79, weak below 55.

Stated limitationsHomepages, not articles, so editorial pages may score higher. The 48% yield favours sites with lighter bot protection. The weights are the author’s judgement, not a published standard.

3%8 of 243 homepages scored 80 or more: structurally quotable.

Strong, 80+
83.3%
Workable, 55–79
7832.1%
Weak, under 55
15764.6%
One square per scored homepage, n = 243Median score 45 / 100

Study 03Published 27 Jul 2026

We scanned 148 businesses in 20 cities. Only 16% sign their work.

Structured data is close to universal at 85.1%. Declaring who wrote a page and when sits at 16.2%, and market maturity does not predict it: Chennai leads, six markets score zero.

Ashish KhanSEO Specialist, MoxSEO

Sample
148 professional service businesses in 20 cities across seven countries, found through Google Places, six to nine per city. Non-resolving domains excluded, not substituted.
Instrument
Study 02’s nine weighted checks on each live homepage, plus robots.txt parsed for 14 AI crawlers.
Licence
Dataset published under CC BY 4.0.

Stated limitationsSmall per-city samples make city percentages indicative. Homepages, not articles. Selection favours firms with strong review profiles, likely biasing scores upward.

16%declare both an author and a date on the homepage: 16.2% of 148.

  1. Chennai43%
  2. Bengaluru38%
  3. Kolkata33%
  4. Ahmedabad33%
  5. Toronto29%
  6. Singapore29%
  7. Gurgaon22%
  8. Mumbai17%
  9. London17%
  10. Delhi13%
  11. Jaipur13%
  12. New York13%
  13. Pune11%
  14. Sydney11%
  15. Lucknow0%
  16. Indore0%
  17. Noida0%
  18. Bhopal0%
  19. Chandigarh0%
  20. Dubai0%
Median score, all 148
68
India, 103 sites
64
International, 45
73
Six to nine sites per city: read city figures as indicativeSource: Study 03 dataset

Method first, mistakes in public

  • A specific question, a documented sample

    Each study states its question, sample, collection date, scoring rules, exclusions and known limitations. Global studies draw from the Tranco top-1M list; local studies find businesses per city through Google Places. Infrastructure domains are filtered out by a disclosed pattern.

  • Measured the way a visitor arrives

    Pages and robots.txt files are fetched with a standard browser user-agent. Homepages are scored, and every study says so.

  • Judgement labelled as judgement

    The nine check weights are ours, not a published standard. Each study states that, and the raw per-check results ship in the dataset so you can re-weight them.

  • Observation, inference and causality kept apart

    A study can show prevalence, co-occurrence and access settings in its sample. It may suggest relationships worth testing. It cannot prove a platform’s ranking logic or that one change will cause visibility. The methodology labels each.

  • Every row published, data handled carefully

    Each study links a CSV with one row per site; the two global files keep every domain attempted. Research uses public information, minimises personal data and documents its transformations; see our privacy policy.

  • From finding to production test

    Findings feed client work through our operating model and AI SEO service. Compare them with real constraints in the Miracuves catalogue project and the case studies.

  • Named people own the evidence

    Studies carry named authors. Evidence standards sit with the founder; the specialist roles that turn a question into inspection and measurement are on the team page.

Corrections, as documented4 entries
  1. Study 01sampling

    Raw Tranco ranks were dominated by infrastructure, CDN and DNS domains rather than websites.

    excluded by pattern before sampling

  2. Study 01collection

    At 175 parallel requests, google.com appeared unreachable. At 25 to 40 it fetched normally.

    first pass discarded, re-run in chunks of 40

  3. Study 02scoring

    The nine checks summed to 110 points, not 100. Caught mid-analysis.

    every reported figure normalised to 100

  4. Study 03reporting

    The write-up said eight checks and five countries; the data supports nine scored checks and seven countries.

    corrected 19 Oct 2026, no figure changed

Each entry is stated in the study it affects.

Written from the data

All articles on the blog

Research questions, answered

What research does MoxSEO publish?

MoxSEO publishes original studies on how websites look to AI systems: which AI crawlers they let in (506 top sites), whether a page offers an answer an AI can lift and credit (243 homepages scored) and whether local businesses sign their work (148 businesses in 20 cities). Each study ships its method, limitations and a CSV.

How are the study samples chosen?

Global studies draw from the Tranco top-1M list, a research-oriented ranking designed to resist manipulation, with infrastructure, CDN and DNS domains removed by a disclosed pattern. The city study found six to nine professional service businesses per city through Google Places. Drop-outs are reported, not replaced.

Can I download and reuse the data?

Yes. Every study links a CSV with one row per site: 845 rows for the crawler study, 506 for the extractability study and 148 for the city study, 1,499 rows in total. The city-market dataset is licensed CC BY 4.0. A link back is appreciated.

What can these studies not prove?

They show patterns in a defined sample at one point in time. They cannot prove how any platform ranks or cites pages, and they cannot guarantee that one change will cause visibility. Treat a finding as a question to test on your own site, not a universal rule.

How should I use a study on my own site?

Pick the pattern closest to your question, check whether it exists on your own rendered pages, decide the expected benefit, make the smallest credible change, validate it in production and measure an outcome that fits. Research informs the decision; your site provides the evidence.

Who writes the studies?

Ashish Khan authored the AI crawler access and city-market studies; Sakshi Kumari authored the AI answer extractability study. Both are SEO specialists who run MoxSEO, and every study carries a named byline, publication and update dates, and a link to the corrections log.

How do I report an error in a study?

Email editorial@moxseo.com with the page URL, the exact statement, why it may be wrong and your evidence. Material errors are fixed and recorded in the public corrections log with the date and whether the conclusion changed.

Where would your site land? Ask the people who ran the studies.

A free 30-minute call with Sakshi Kumari or Ashish Khan, the specialists who run MoxSEO and authored this research. Bring the question your team needs to answer; they will walk your homepage through the same nine checks and your robots.txt through the same 14 crawlers.

Book a free strategy call Or email contact@moxseo.com

A call is not an engagement. Plans start at $699 a month, and we never guarantee rankings.

The nine checksStudy 02, n = 243
The nine structural checks, their raw weight and how many top-site homepages passed each
CheckWeightPass rate barPass
Author and dates declared105.8%
Question-form headings1016.9%
Direct answer near the top2020.6%
Cites two or more external sources1044%
Lists or tables1544.9%
Exactly one H11046.1%
Structured data1053.1%
Three or more H2 sections1558.4%
No paragraph over 120 words1098.8%
110 raw points, normalised to 100.Run them on your page