llms.txt is a proposed convention for helping large language models understand a website more efficiently. Published as a Markdown file at /llms.txt, it summarizes what the organization does and highlights the pages, documentation, research, or resources that matter most. Instead of functioning like a sitemap that merely lists URLs, it adds short descriptions that explain the purpose and relevance of each selected page.
However, llms.txt is still a proposal rather than an established web standard. There is no reliable evidence that it improves rankings, AI citations, or visibility in assistant-generated answers. It is worth understanding and may justify around twenty minutes of implementation time, but it should not take priority over crawlability, clear page structure, useful content, internal linking, accurate authorship, and other improvements with demonstrated benefits.
What the file is
A plain Markdown file at /llms.txt, structured as a title, a short summary, and lists of links with one-line descriptions. It is written for a machine that wants an overview of your site without crawling all of it.
The idea borrows from robots.txt in placement and from a sitemap in purpose, but it differs from both. robots.txt controls access. A sitemap lists URLs. llms.txt describes meaning.
Who actually reads it
This is where honesty is required: adoption is limited and unevenly documented. No major assistant has committed to honouring it as a standard, and it is not part of any specification body.
What is measurable is that it costs almost nothing to publish and does no harm. That asymmetry is the entire argument for it, and it is a reasonable argument — but it is not the same as evidence it works.
Be sceptical of anyone claiming ranking benefits from llms.txt. There is no data supporting that, including ours.
We scanned 845 domains from the Tranco top-1M and found llms.txt adoption negligible enough that it did not warrant its own finding in the study. That is not an argument against publishing one. It is an argument against expecting anything measurable from it yet.
What a good one contains
If you publish one, make it genuinely useful rather than a second sitemap.
- A one-paragraph description of what your organisation actually does
- Your most important pages, with a sentence explaining each
- Documentation, datasets or research, if you publish any
- What you do not want used, stated plainly
Our llms.txt generator and validator checks an existing file against the proposed structure, or builds a starting point from your sitemap.
What to do instead, if you only do one thing
The things that demonstrably affect whether an assistant can use your content are more boring and more effective: a direct answer near the top of the page, clear heading structure, declared authorship and dates, and crawler access that does not accidentally block the assistants you want.
We scored 243 homepages against those factors and found only 3% structurally quotable. That gap is worth far more attention than a file almost nothing reads yet.
Publish llms.txt if you like. Fix the page structure first.
What a well-formed file looks like
The proposal specifies Markdown with a particular shape: an H1 with the site name, a blockquote summary, then H2 sections containing link lists with one-line descriptions.
The discipline that makes it useful is the descriptions. A list of URLs is a sitemap. A list of URLs each explained in a sentence is something a model can actually use to decide what is relevant.
Keep it short. A file listing four hundred pages defeats the purpose, which is to give an overview rather than an inventory. Twenty to fifty well-chosen entries is far more useful than everything you have.
How this differs from robots.txt and sitemaps
The three are frequently confused because they share a location convention and nothing else.
- robots.txt — access control. Which crawlers may fetch which paths. Honoured by convention and long established.
- sitemap.xml — discovery. Which URLs exist and when they changed. A formal, widely supported protocol.
- llms.txt — description. What the site is about and which parts matter. A proposal, not a standard.
Publishing llms.txt does not affect crawling and does not replace either of the others. If you have to choose where to spend attention, robots.txt is the one that demonstrably changes behaviour today.
The measurable version of this problem
If your actual goal is being usable by AI assistants, there are factors with evidence behind them. We scored 243 homepages against nine of them and found only 3% structurally quotable.
- A direct answer near the top — the single heaviest factor in our scoring
- Clear heading structure — one H1, no skipped levels
- Declared authorship and dates — failed by 84% of the businesses we scanned
- Crawler access — check you are not blocking assistants you want citing you
Every one of those is measurable today with our extractability checker. Fix them first. Publish llms.txt afterwards if you still want to.
Make Your Website Easier for Search Engines and AI to Understand
Publishing an llms.txt file is a useful experiment, but it should be part of a broader technical and content strategy. MoxSEO can review your site structure, crawler access, headings, internal links, authorship signals, and content extractability to identify the improvements most likely to strengthen search and AI visibility.
You can request a free SEO audit to uncover technical and structural gaps or schedule an SEO consultation for a focused plan based on your website’s current setup.
Sources and further reading
Frequently asked questions
Will llms.txt improve my rankings?
There is no evidence that it does, and anyone claiming otherwise is ahead of the data.
Does it replace robots.txt?
No. They do different jobs. robots.txt controls access; llms.txt describes content. Keep both.
Can I block AI training with it?
Not reliably. Use robots.txt for access control, which is the mechanism crawlers actually honour.
Is it worth publishing at all?
It costs twenty minutes and does no harm. Just do not expect a measurable return.
Ashish Khan is an SEO Specialist at MoxSEO with expertise in keyword research, on-page optimization, technical SEO, content strategy, and link building. He focuses on improving website visibility, strengthening search performance, and helping businesses attract relevant organic traffic. By combining competitor analysis, SEO audits, and data-driven optimization, Ashish supports sustainable ranking growth and stronger digital presence.
