robots.txt says what crawlers may not read. llms.txt says what is worth reading. Check whether yours is valid, or generate a starter file from your sitemap in one click.
Language models do not browse the way people do. They arrive with a question, pull a handful of pages, and assemble an answer. If nothing tells them how your site is organised or which pages carry authority, they guess — and they guess from navigation labels and page titles.
A crawler sees a flat list of URLs. It cannot tell that your pricing page matters more than a 2023 changelog entry, or that six of your posts are one cluster. llms.txt makes that hierarchy explicit.
The summary blockquote under the title is the single highest-leverage line in the file. It is often what a model repeats when someone asks what your company does.
Tag archives, pagination and thin utility pages get crawled alongside your best work. A curated file points models at what you would actually want quoted.
Exactly one # Heading at the top. More than one confuses parsers about which is canonical; zero leaves the file unattributed.
One > line directly under the title, describing what the site offers in a sentence. This is the part most generated files omit and the part that matters most.
## Docs, ## Products, ## Guides — grouping that mirrors how a person would ask about your site, not how your CMS happens to store it.
Each link followed by a short description of what the page covers. The annotation is what lets a model choose between two similar URLs without fetching both.
Enter a domain. If an llms.txt exists we check it against the spec and show you a preview. If it does not, we build a starter file from your sitemap that you can edit and upload.
Reads only public files. Nothing is stored.
Title, summary, sections and annotated links all present. Assistants can parse your structure without inference.
Typically a missing summary line or no section grouping. Functional, but you are leaving the most valuable signal on the table.
Nothing curated for AI systems. Generate a starter file below, edit the summary, group the sections, and upload it to your site root.
Adding llms.txt does not make you rank, does not guarantee citation, and is not confirmed as a signal by any major model provider. Anyone selling it as a ranking lever is overstating it.
It costs almost nothing, it removes ambiguity about your positioning, and adoption is growing quickly among documentation sites and SaaS products. Cheap, reversible, and it forces you to articulate what your site is for — which is useful regardless.
At your site root, served as plain text: yourdomain.com/llms.txt. Same location convention as robots.txt.
No, they do opposite jobs. robots.txt restricts access; llms.txt curates and describes. You want both, and they should not contradict each other — check your crawler access with our AI crawler tool.
Usually not. Auto-generated files tend to dump every URL without grouping or annotation, and most omit the summary blockquote entirely. A generated file is a reasonable starting point that still needs a human pass.
Curated, not exhaustive. A few dozen well-annotated links beats several hundred bare URLs. If your file is hundreds of kilobytes, it is a sitemap wearing a costume.
Whenever your site structure changes meaningfully. Watch for staleness — a file listing pages you deleted months ago actively misleads the systems reading it.
If writing the summary line is hard, that is usually a positioning problem rather than a file-format problem — and it is worth solving either way.