AI crawler policy study
We checked the robots.txt of 506 widely visited websites against 14 AI crawlers and found sites block training crawlers roughly twice as often as retrieval crawlers.
Search engineering,
end to end.
26 services across SEO, digital marketing and web development, run by the people who sign the work.
All services
+340%
organic patient growth for Dermacian, a dermatology clinic
Markets where
accuracy is the job.
Each market has its own review standards, page types and measurement limits. We plan around them from day one.
All industries
Healthcare & MedTech
Clinical review, accurate provider facts, proportionate claims.
FinTech
Fees, rates and the regulated entity stated plainly and dated.
B2B SaaS
Category, use-case, integration and security pages with distinct jobs.
Ecommerce & marketplaces
Rules for filters, stock states, sellers and reviews.
Legal & law firms
Jurisdiction, attorney involvement and advertising rules.
24 free tools.
No signup.
Can AI search quote your page? Paste a URL and the extractability checker scores it against nine structural checks.
All 24 toolsAI search 8
Technical 9
Entity 4
Local and global 3
Named people.
Published method.
Sakshi Kumari and Ashish Khan run MoxSEO's search work and sign the research they publish.
Meet the teamAI SEO services make your pages easy for answer engines to reach, quote and attribute. We open crawler access, rewrite priority pages around direct answers, connect your entities with structured data and publish evidence worth citing, so ChatGPT, Perplexity and Google AI Overviews can name you. Plans start at $699 a month.
Reviewed by Sakshi Kumari, SEO specialist at MoxSEO. Last reviewed 19 October 2026.
robots.txtUser-agent: * Disallow: / (AI bots)
Intendedblock GPTBot · allow OAI-SearchBot
AI SEO is not a replacement for technical SEO. Answer engines still depend on accessible pages, clear structure and credible sources. Generative engine optimisation adds a second test: your strongest facts must be easy to extract, understand and attribute.
User-agent: * DisallowIn our study of 506 top sites, sites blocked AI training crawlers roughly twice as often as retrieval crawlers. A blanket rule blocks both, including the ones that send citations and referral traffic.
headline → hero → navOnly 20.6% of the 243 sites in our extractability benchmark put a direct answer near the top. The median opening paragraph was 18 words, too short to hold an answer.
author datePublished dateModifiedOnly 5.8% of those sites declared both an author and dates. Systems that cite prefer content they can attribute and date.
Organization sameAs @idWhen the organisation, its services, people and research aren't connected through stable identifiers, models misstate or omit what you do.
dataset method studyGeneric content restates what models already know. Original data, methods and tools give them a reason to name you as the source.
citations AI referralsWithout a baseline of priority questions and cited pages, AI visibility work turns into guesswork and screenshots.
Five areas we implement, plus the audit that decides their order. Each one changes something a crawler or answer engine can observe.
| Deliverable | What it covers | Touches |
|---|---|---|
| AI visibility diagnostic | Priority questions mapped, crawler access and rendering inspected, citations benchmarked, and the commercial pages with the largest opportunity identified. | prompt setbaseline |
| AI crawler and rendering controls | Audit of robots.txt, response headers, rendered HTML and JavaScript dependencies for Google, OpenAI, Anthropic, Perplexity and other relevant crawlers, with a policy that separates training from retrieval. | robots.txtheadersrendered HTML |
| Answer-ready commercial content | Priority pages rewritten around explicit definitions, decision criteria, comparison data and concise answers that stay useful to human buyers. | opening answerquestion H2stables |
| Knowledge graph and schema alignment | The organisation, services, leadership, research and supporting assets connected through stable identifiers and appropriate structured data. | OrganizationPersonsameAs |
| Original research and citation assets | Datasets, methodology pages, tools and studies that add information competitors can't reproduce with generic content. | Datasetmethodology |
| Search and AI visibility reporting | Cited answers across agreed questions, source attribution, qualified AI-referral sessions, assisted conversions and the commercial pages receiving that demand. | citation logAI referrals |
The sequence changes with your platform and market, but implementation starts with the assets that influence commercial discovery. Measurement runs from day one.
Map priority questions, audit AI crawler access, inspect rendering, benchmark citations and find the commercial pages with the largest opportunity.
Refactor answer blocks, clarify service evidence, connect entity schema and ship technical changes into the agreed production workflow.
Publish differentiated evidence, measure citation and referral changes, review assisted conversions and expand the patterns that produce qualified demand.
AI answer presence, source attribution, qualified referral sessions and assisted conversions against the day-one baseline.
The plan is typical and is confirmed in the written scope. No one can guarantee how an external model will answer.
We publish the method, sample and limits.Each study links its methodology, and changes are recorded in a public corrections log.
We checked the robots.txt of 506 widely visited websites against 14 AI crawlers and found sites block training crawlers roughly twice as often as retrieval crawlers.
We scored 243 leading websites on whether their content could be lifted cleanly and attributed. The median score was 45 out of 100, and only 3.3% met the strongest threshold.
Across 148 businesses in 20 cities, only 16.2% declared both an author and a date: a widespread attribution gap.
Published starting points, not automatic quotes. Final scope depends on the site, markets, access, technical condition and content requirements, and a written proposal sets the fee.
A focused starting scope for one clear priority and a smaller operating surface.
A broader program for teams that need recurring execution and coordinated improvement.
A custom operating scope for larger sites, markets, catalogues or stakeholder groups.
No plan includes guaranteed placement in any AI-generated answer.
GEO is new enough that anyone can claim it. Four promises a responsible agency won't make, and what to ask instead.
No provider controls how an external model answers. Ask for a documented baseline and the changes they will measure instead.
A score is only useful if you can see how it's calculated. Ask for the checks, their weights and the sample behind them.
More pages don't make content quotable. Ask how they'll restructure the pages you already have around direct answers.
Untraceable metrics and unnamed clients aren't evidence. Ask for case studies and research you can inspect.
Five free tools built from our research. No sign-up.
Question not here? Strategy calls are with Sakshi Kumari or Ashish Khan, the SEO specialists who run MoxSEO.
contact@moxseo.comAI SEO services make a website easier for AI answer engines such as ChatGPT, Perplexity and Google AI Overviews to crawl, understand, quote and attribute. The work covers crawler access, answer-ready page structure, entity and schema clarity, original evidence and measurement of citations and AI referrals.
Generative engine optimization (GEO) improves the likelihood that an organisation's content is understood, retrieved and attributed by AI answer systems. It combines technical access, extractable content, entity clarity and credible evidence.
No. Crawlability, indexing, internal linking, page quality and authority remain essential. GEO extends that foundation so important facts can also be selected and cited in generated answers.
Decide crawler by crawler. Training crawlers such as GPTBot and ClaudeBot gather model training data; retrieval crawlers such as OAI-SearchBot and Claude-SearchBot fetch pages so they can be cited. Many sites block training but allow retrieval. A blanket rule blocks both.
No responsible provider can guarantee how an external model will answer. MoxSEO documents the baseline, implements controllable improvements and measures observable changes without presenting probabilistic systems as guaranteed outcomes.
Measurement can include citation presence across agreed questions, attributed source pages, AI-referral sessions, assisted conversions, non-brand search demand and the performance of priority commercial assets.
Software automates parts of the work, such as keyword research, audits and drafting, and you operate it yourself. An agency provides the strategy, implementation and ongoing measurement, usually using such tools. If you have an in-house team with SEO skills, software may be enough; if you need someone to plan and do the work, an agency or specialist fits better.
Our published plans start at $699/month (AI Search Starter), $1,299/month (Enterprise GEO Scale) and $2,499/month (Global Category Program). The final fee depends on the site, markets and content work, and is set in a written proposal.
Bring your priority markets, commercial pages and current search data. In a free 30-minute call, Sakshi Kumari or Ashish Khan, the specialists who run MoxSEO, identify the first technical and content decisions worth making.
A call is not an engagement. AI SEO plans start at $699 a month, and we never guarantee citations or rankings.