---
title: llms.txt Explained: Is It Worth Your Time Yet?
description: What the file does, who reads it, and an honest view of its current adoption.
url: https://moxseo.com/blogs/llms-txt-explained
date_modified: 2026-10-28
author: Ashish Khan
language: en_US
---

llms.txt is a proposed convention for helping large language models understand a website more efficiently. Published as a Markdown file at `/llms.txt`, it summarizes what the organization does and highlights the pages, documentation, research, or resources that matter most. Instead of functioning like a sitemap that merely lists URLs, it adds short descriptions that explain the purpose and relevance of each selected page.

However, llms.txt is still a proposal rather than an established web standard. There is no reliable evidence that it improves rankings, AI citations, or visibility in assistant-generated answers. It is worth understanding and may justify around twenty minutes of implementation time, but it should not take priority over crawlability, clear page structure, useful content, internal linking, accurate authorship, and other improvements with demonstrated benefits.

## What the file is

A plain Markdown file at `/llms.txt`, structured as a title, a short summary, and lists of links with one-line descriptions. It is written for a machine that wants an overview of your site without crawling all of it.

The idea borrows from [robots.txt](/tools/robots-txt-tester/) in placement and from a sitemap in purpose, but it differs from both. robots.txt controls access. A sitemap lists URLs. llms.txt describes meaning.

The proposal was published by Jeremy Howard in September 2024 and is maintained at [llmstxt.org](https://llmstxt.org/) (checked October 2026). It describes itself as a proposal to standardise a file that helps agents use a website, and says it complements sitemap.xml and robots.txt rather than replacing them.

## Who actually reads it

This is where honesty is required: adoption is limited and unevenly documented. No major assistant has committed to honouring it as a standard, and it is not part of any specification body.

What is measurable is that it costs almost nothing to publish and does no harm. That asymmetry is the entire argument for it, and it is a reasonable argument — but it is not the same as evidence it works.

Be sceptical of anyone claiming ranking benefits from llms.txt. There is no data supporting that, including ours.

Our [crawler blocking study](/ai-crawler-blocking-study/) scanned 845 domains from the Tranco top-1M, and llms.txt was not one of its findings. That is not an argument against publishing one. It is an argument against expecting anything measurable from it yet. Related research: our [AI crawler blocking study](/ai-crawler-blocking-study/).

## What a good one contains

If you publish one, make it genuinely useful rather than a second sitemap.

- A one-paragraph description of what your organisation actually does
- Your most important pages, with a sentence explaining each
- Documentation, datasets or research, if you publish any
- What you do not want used, stated plainly

Our [llms.txt generator and validator](https://moxseo.com/tools/llms-txt-generator/) checks an existing file against the proposed structure, or builds a starting point from your sitemap.

## How do you create an llms.txt file, step by step?

Write a Markdown file with your site name as an H1, a short blockquote summary, and H2 sections of links with one-line notes, then save it as `llms.txt` at your site root and check it loads as plain text. The H1 is the only required element in the proposal.

1. **Choose the pages.** Pick the 20 to 50 pages that best explain what you do: core services or products, pricing, documentation, research, contact.
2. **Write the summary.** Two or three sentences in a blockquote saying who you are and what the site covers.
3. **Group the links.** Use H2 headings for groups, and list each page as `[Page name](URL): what it contains`.
4. **Mark secondary items.** By the proposal’s convention, an H2 named “Optional” holds links an agent can skip when it needs a shorter context.
5. **Publish and test.** Upload it to `/llms.txt`, open it in a browser, and confirm it returns a 200 status as plain text, not a redirect or an HTML page.
6. **Keep it current.** Review it when you add or retire key pages.

### A hypothetical example

This sample uses a made-up business and placeholder URLs to show the shape of the file:

```
# Example Plumbing Co

> Example Plumbing Co is a licensed plumbing service covering one fictional city. This file lists the pages that best describe our services and prices.

## Services

- [Emergency repairs](https://example.com/emergency-repairs/): What we fix, response areas and call-out terms
- [Boiler servicing](https://example.com/boiler-servicing/): Annual service checklist and what is included

## Company

- [About](https://example.com/about/): Who we are, qualifications and service area
- [Contact](https://example.com/contact/): Phone, hours and booking form

## Optional

- [Blog](https://example.com/blog/): Maintenance guides and seasonal advice
```

## What to do instead, if you only do one thing

![AI-readable content structure showing clear headings, authorship and crawler access before llms.txt.](https://moxseo.com/wp-content/uploads/2026/08/ai-readable-content-structure-before-llms-txt-1024x683.webp)Image Source: AI-generated visual by MoxSEO  
 

The things that demonstrably affect whether an assistant can use your content are more boring and more effective: a direct answer near the top of the page, clear heading structure, declared authorship and dates, and crawler access that does not accidentally block the assistants you want.

We scored 243 homepages against those factors and found only 3% structurally quotable. That gap is worth far more attention than a file almost nothing reads yet.

Publish llms.txt if you like. Fix the page structure first.

## What a well-formed file looks like

The proposal specifies Markdown with a particular shape: an H1 with the site name, a blockquote summary, then H2 sections containing link lists with one-line descriptions.

The discipline that makes it useful is the descriptions. A list of URLs is a sitemap. A list of URLs each explained in a sentence is something a model can actually use to decide what is relevant.

Keep it short. A file listing four hundred pages defeats the purpose, which is to give an overview rather than an inventory. Twenty to fifty well-chosen entries is far more useful than everything you have.

## How this differs from robots.txt and sitemaps

The three are frequently confused because they share a location convention and nothing else.

- **robots.txt** — access control. Which crawlers may fetch which paths. Honoured by convention and long established.
- **sitemap.xml** — discovery. Which URLs exist and when they changed. A formal, widely supported protocol.
- **llms.txt** — description. What the site is about and which parts matter. A proposal, not a standard.

Publishing llms.txt does not affect crawling and does not replace either of the others. If you have to choose where to spend attention, robots.txt is the one that demonstrably changes behaviour today.

To see which AI crawlers your robots.txt currently allows or blocks, run it through our [AI crawler access checker](/tools/ai-crawler-access-checker/).

## Common llms.txt mistakes

- **Pasting the whole sitemap.** It becomes an inventory, not an overview.
- **Links without descriptions.** The notes are what make the file more useful than a sitemap.
- **Linking to redirected or broken URLs.** Check each link returns a 200.
- **Expecting it to block crawlers.** Use robots.txt for access control.
- **Marketing copy in the summary.** A plain factual description is easier to use than slogans.

## The measurable version of this problem

If your actual goal is being usable by AI assistants, there are factors with evidence behind them. We scored 243 homepages against nine of them and found only 3% structurally quotable.

- **A direct answer near the top** — the single heaviest factor in our scoring
- **Clear heading structure** — one H1, no skipped levels
- **Declared authorship and dates** — failed by 84% of the businesses we scanned
- **Crawler access** — check you are not blocking assistants you want citing you

Every one of those is measurable today with our [extractability checker](https://moxseo.com/tools/ai-answer-extractability-checker/). Fix them first. Publish llms.txt afterwards if you still want to. Related research: our [AI answer extractability study](/ai-answer-extractability-study/). Our [AI SEO service](/services/seo/ai/) applies it to client sites.

## Make Your Website Easier for Search Engines and AI to Understand

Publishing an llms.txt file is a useful experiment, but it should be part of a broader technical and content strategy. MoxSEO can review your site structure, crawler access, headings, internal links, authorship signals, and content extractability to identify the improvements most likely to strengthen search and AI visibility.

You can [request a free SEO audit](https://moxseo.com/get-free-quote/) to uncover technical and structural gaps or [schedule an SEO consultation](https://moxseo.com/schedule-consultation/) for a focused plan based on your website’s current setup.

## Is llms.txt a standard every website needs?

No. llms.txt is a community proposal published at [llmstxt.org](https://llmstxt.org/), not a standard issued by a standards body, and no major AI provider has committed to reading it as a ranking or citation input. Google’s guidance for AI features in Search says pages need no special markup or files to appear in AI Overviews; being indexed and eligible for a snippet is what counts ([Google Search Central](https://developers.google.com/search/docs/appearance/ai-features), as of October 2026). Articles that call llms.txt a requirement for every site in 2026 overstate what is known.

That does not make the file useless. For documentation sites, developer products and large knowledge bases, a short, curated map of your most important pages is a cheap way to help any tool that does read it, and it forces a useful editorial question: which twenty pages actually explain what you do?

## Does llms.txt control AI access or crawl load?

No. llms.txt is a reading guide, not an access-control file. It cannot block a crawler, and it does not change how often bots visit or how much server capacity they use. If you want to decide which AI systems may fetch your pages, use robots.txt rules for each crawler’s published user agent, such as GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and the Google-Extended product token for Gemini training. Check each provider’s own crawler documentation before you rely on a name.

- **To allow or block AI crawlers:** robots.txt user-agent rules. Our [AI Crawler Access Checker](/tools/ai-crawler-access-checker/) shows what your current rules allow.
- **To reduce load from aggressive bots:** rate limiting at your CDN or server, not llms.txt.
- **To help AI answer engines quote you accurately:** clear, well-structured pages. See our [AI answer extractability study](/ai-answer-extractability-study/) for what made 243 sites quotable or not.

## How do you keep an llms.txt file current?

A stale llms.txt that points to moved or deleted pages is worse than none. If you publish one, treat it like your sitemap and check it whenever you release site changes:

1. Confirm `/llms.txt` returns HTTP 200 as plain text.
2. Check that every linked URL resolves without a redirect or 404.
3. Update the summary and page list when you launch, rename or retire a key page.
4. Review it at least quarterly, and validate it with our free [llms.txt Generator and Validator](/tools/llms-txt-generator/).

Teams with a deployment pipeline can automate the first two checks as a release step, the same way they test that robots.txt and the [XML sitemap](/tools/xml-sitemap-validator/) are reachable.

**Related:** [AI SEO and generative engine optimisation services](/services/seo/ai/), [AI SEO packages](/packages/seo/ai-seo-packages/), the [AI Answer Extractability Checker](/tools/ai-answer-extractability-checker/) and the [llms.txt Generator](/tools/llms-txt-generator/).

## Sources and further reading

- [The llms.txt proposal](https://llmstxt.org/)
- [Robots Exclusion Protocol, RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)

## Frequently asked questions

### Will llms.txt improve my rankings?

There is no evidence that it does, and anyone claiming otherwise is ahead of the data.

### Does it replace robots.txt?

No. They do different jobs. robots.txt controls access; llms.txt describes content. Keep both.

### Can I block AI training with it?

Not reliably. Use robots.txt for access control, which is the mechanism crawlers actually honour.

### Is it worth publishing at all?

It costs twenty minutes and does no harm. Just do not expect a measurable return.

### Where does the llms.txt file go?

At the root of your site, for example example.com/llms.txt. The proposal also allows the file in a subpath, such as a documentation folder, where it covers URLs under that path.

### What must an llms.txt file contain?

Under the proposal, the only required element is an H1 with the site or project name. A short blockquote summary and H2 sections of linked pages with notes are the usual additions.

### Who created llms.txt?

It was proposed by Jeremy Howard and published at llmstxt.org in September 2024. It is a community proposal, not a standard issued by a standards body.

### Should WordPress sites do anything special for llms.txt?

No. Create the file and upload it to the site root, or use a generator, then confirm it loads as plain text at the right address. Do not install a plugin only for this unless you will keep it maintained.
