
llms.txt is a proposed convention for helping large language models understand a website more efficiently. Published as a Markdown file at /llms.txt, it summarizes what the organization does and highlights the pages, documentation, research, or resources that matter most. Instead of functioning like a sitemap that merely lists URLs, it adds short descriptions that explain the purpose and relevance of each selected page.
However, llms.txt is still a proposal rather than an established web standard. There is no reliable evidence that it improves rankings, AI citations, or visibility in assistant-generated answers. It is worth understanding and may justify around twenty minutes of implementation time, but it should not take priority over crawlability, clear page structure, useful content, internal linking, accurate authorship, and other improvements with demonstrated benefits.
What the file is
A plain Markdown file at /llms.txt, structured as a title, a short summary, and lists of links with one-line descriptions. It is written for a machine that wants an overview of your site without crawling all of it.
The idea borrows from robots.txt in placement and from a sitemap in purpose, but it differs from both. robots.txt controls access. A sitemap lists URLs. llms.txt describes meaning.
The proposal was published by Jeremy Howard in September 2024 and is maintained at llmstxt.org (checked October 2026). It describes itself as a proposal to standardise a file that helps agents use a website, and says it complements sitemap.xml and robots.txt rather than replacing them.
Who actually reads it
This is where honesty is required: adoption is limited and unevenly documented. No major assistant has committed to honouring it as a standard, and it is not part of any specification body.
What is measurable is that it costs almost nothing to publish and does no harm. That asymmetry is the entire argument for it, and it is a reasonable argument — but it is not the same as evidence it works.
Be sceptical of anyone claiming ranking benefits from llms.txt. There is no data supporting that, including ours.
Our crawler blocking study scanned 845 domains from the Tranco top-1M, and llms.txt was not one of its findings. That is not an argument against publishing one. It is an argument against expecting anything measurable from it yet. Related research: our AI crawler blocking study.
What a good one contains
If you publish one, make it genuinely useful rather than a second sitemap.
- A one-paragraph description of what your organisation actually does
- Your most important pages, with a sentence explaining each
- Documentation, datasets or research, if you publish any
- What you do not want used, stated plainly
Our llms.txt generator and validator checks an existing file against the proposed structure, or builds a starting point from your sitemap.
How do you create an llms.txt file, step by step?
Write a Markdown file with your site name as an H1, a short blockquote summary, and H2 sections of links with one-line notes, then save it as llms.txt at your site root and check it loads as plain text. The H1 is the only required element in the proposal.
- Choose the pages. Pick the 20 to 50 pages that best explain what you do: core services or products, pricing, documentation, research, contact.
- Write the summary. Two or three sentences in a blockquote saying who you are and what the site covers.
- Group the links. Use H2 headings for groups, and list each page as
[Page name](URL): what it contains. - Mark secondary items. By the proposal’s convention, an H2 named “Optional” holds links an agent can skip when it needs a shorter context.
- Publish and test. Upload it to
/llms.txt, open it in a browser, and confirm it returns a 200 status as plain text, not a redirect or an HTML page. - Keep it current. Review it when you add or retire key pages.
A hypothetical example
This sample uses a made-up business and placeholder URLs to show the shape of the file:
# Example Plumbing Co
> Example Plumbing Co is a licensed plumbing service covering one fictional city. This file lists the pages that best describe our services and prices.
## Services
- [Emergency repairs](https://example.com/emergency-repairs/): What we fix, response areas and call-out terms
- [Boiler servicing](https://example.com/boiler-servicing/): Annual service checklist and what is included
## Company
- [About](https://example.com/about/): Who we are, qualifications and service area
- [Contact](https://example.com/contact/): Phone, hours and booking form
## Optional
- [Blog](https://example.com/blog/): Maintenance guides and seasonal adviceWhat to do instead, if you only do one thing

The things that demonstrably affect whether an assistant can use your content are more boring and more effective: a direct answer near the top of the page, clear heading structure, declared authorship and dates, and crawler access that does not accidentally block the assistants you want.
We scored 243 homepages against those factors and found only 3% structurally quotable. That gap is worth far more attention than a file almost nothing reads yet.
Publish llms.txt if you like. Fix the page structure first.
What a well-formed file looks like
The proposal specifies Markdown with a particular shape: an H1 with the site name, a blockquote summary, then H2 sections containing link lists with one-line descriptions.
The discipline that makes it useful is the descriptions. A list of URLs is a sitemap. A list of URLs each explained in a sentence is something a model can actually use to decide what is relevant.
Keep it short. A file listing four hundred pages defeats the purpose, which is to give an overview rather than an inventory. Twenty to fifty well-chosen entries is far more useful than everything you have.
How this differs from robots.txt and sitemaps
The three are frequently confused because they share a location convention and nothing else.
- robots.txt — access control. Which crawlers may fetch which paths. Honoured by convention and long established.
- sitemap.xml — discovery. Which URLs exist and when they changed. A formal, widely supported protocol.
- llms.txt — description. What the site is about and which parts matter. A proposal, not a standard.
Publishing llms.txt does not affect crawling and does not replace either of the others. If you have to choose where to spend attention, robots.txt is the one that demonstrably changes behaviour today.
To see which AI crawlers your robots.txt currently allows or blocks, run it through our AI crawler access checker.
Common llms.txt mistakes
- Pasting the whole sitemap. It becomes an inventory, not an overview.
- Links without descriptions. The notes are what make the file more useful than a sitemap.
- Linking to redirected or broken URLs. Check each link returns a 200.
- Expecting it to block crawlers. Use robots.txt for access control.
- Marketing copy in the summary. A plain factual description is easier to use than slogans.
The measurable version of this problem
If your actual goal is being usable by AI assistants, there are factors with evidence behind them. We scored 243 homepages against nine of them and found only 3% structurally quotable.
- A direct answer near the top — the single heaviest factor in our scoring
- Clear heading structure — one H1, no skipped levels
- Declared authorship and dates — failed by 84% of the businesses we scanned
- Crawler access — check you are not blocking assistants you want citing you
Every one of those is measurable today with our extractability checker. Fix them first. Publish llms.txt afterwards if you still want to. Related research: our AI answer extractability study. Our AI SEO service applies it to client sites.
Make Your Website Easier for Search Engines and AI to Understand
Publishing an llms.txt file is a useful experiment, but it should be part of a broader technical and content strategy. MoxSEO can review your site structure, crawler access, headings, internal links, authorship signals, and content extractability to identify the improvements most likely to strengthen search and AI visibility.
You can request a free SEO audit to uncover technical and structural gaps or schedule an SEO consultation for a focused plan based on your website’s current setup.
Is llms.txt a standard every website needs?
No. llms.txt is a community proposal published at llmstxt.org, not a standard issued by a standards body, and no major AI provider has committed to reading it as a ranking or citation input. Google’s guidance for AI features in Search says pages need no special markup or files to appear in AI Overviews; being indexed and eligible for a snippet is what counts (Google Search Central, as of October 2026). Articles that call llms.txt a requirement for every site in 2026 overstate what is known.
That does not make the file useless. For documentation sites, developer products and large knowledge bases, a short, curated map of your most important pages is a cheap way to help any tool that does read it, and it forces a useful editorial question: which twenty pages actually explain what you do?
Does llms.txt control AI access or crawl load?
No. llms.txt is a reading guide, not an access-control file. It cannot block a crawler, and it does not change how often bots visit or how much server capacity they use. If you want to decide which AI systems may fetch your pages, use robots.txt rules for each crawler’s published user agent, such as GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and the Google-Extended product token for Gemini training. Check each provider’s own crawler documentation before you rely on a name.
- To allow or block AI crawlers: robots.txt user-agent rules. Our AI Crawler Access Checker shows what your current rules allow.
- To reduce load from aggressive bots: rate limiting at your CDN or server, not llms.txt.
- To help AI answer engines quote you accurately: clear, well-structured pages. See our AI answer extractability study for what made 243 sites quotable or not.
How do you keep an llms.txt file current?
A stale llms.txt that points to moved or deleted pages is worse than none. If you publish one, treat it like your sitemap and check it whenever you release site changes:
- Confirm
/llms.txtreturns HTTP 200 as plain text. - Check that every linked URL resolves without a redirect or 404.
- Update the summary and page list when you launch, rename or retire a key page.
- Review it at least quarterly, and validate it with our free llms.txt Generator and Validator.
Teams with a deployment pipeline can automate the first two checks as a release step, the same way they test that robots.txt and the XML sitemap are reachable.
Related: AI SEO and generative engine optimisation services, AI SEO packages, the AI Answer Extractability Checker and the llms.txt Generator.
Sources and further reading
Frequently asked questions
Will llms.txt improve my rankings?
There is no evidence that it does, and anyone claiming otherwise is ahead of the data.
Does it replace robots.txt?
No. They do different jobs. robots.txt controls access; llms.txt describes content. Keep both.
Can I block AI training with it?
Not reliably. Use robots.txt for access control, which is the mechanism crawlers actually honour.
Is it worth publishing at all?
It costs twenty minutes and does no harm. Just do not expect a measurable return.
Where does the llms.txt file go?
At the root of your site, for example example.com/llms.txt. The proposal also allows the file in a subpath, such as a documentation folder, where it covers URLs under that path.
What must an llms.txt file contain?
Under the proposal, the only required element is an H1 with the site or project name. A short blockquote summary and H2 sections of linked pages with notes are the usual additions.
Who created llms.txt?
It was proposed by Jeremy Howard and published at llmstxt.org in September 2024. It is a community proposal, not a standard issued by a standards body.
Should WordPress sites do anything special for llms.txt?
No. Create the file and upload it to the site root, or use a generator, then confirm it loads as plain text at the right address. Do not install a plugin only for this unless you will keep it maintained.






