llms.txt: What It Is, How to Write One, and Whether It Helps

What is llms.txt?

llms.txt is a proposed standard for a markdown file at the root of a website that tells AI language models what the site is about and which pages matter most. It was proposed by Jeremy Howard of Answer.AI in September 2024. The file uses a fixed structure: a title, a short summary, and lists of links with one-line notes.

The idea behind it is simple. Web pages are built for browsers. They carry navigation, ads, scripts, and layout that a language model has to strip away before it can read the words. A model also has a limited context window, which is the amount of text it can hold at once. The file skips all that. It hands the model a clean list of your best pages, in a format it reads natively.

The proposal has two companions. The first is markdown copies of your pages, served at the same URL with .md on the end. The second is llms-full.txt, a community convention where you put the full text of your documentation into one large file. Both are optional. Most sites that adopt the idea start with the small index file and stop there.

llms.txt vs robots.txt vs sitemap.xml

People mix these up, so here is the split. robots.txt tells crawlers what they may fetch. A sitemap tells them what exists. The third file tells a language model what is worth reading and what it means. Only the first two are supported by Google.

One practical point follows. If you block an AI crawler in robots.txt, it will not read the file either. The files do not override each other. They stack. Our technical SEO checklist covers robots and sitemap setup if you need to fix those first.

Is llms.txt actually used?

By websites, yes. By the AI companies it was written for, not that anyone has confirmed. That is the whole debate in two sentences.

On the adoption side, documentation platforms such as Mintlify and GitBook generate the file for every site they host. Many developer tool companies publish one. Google's Lighthouse tool now includes a check for the file in its audits for agentic browsing, which shows the idea has some weight at Google's Chrome team.

On the usage side, the evidence points the other way. In 2025, Google's John Mueller said that no AI system he knew of currently used the file. He compared it to the keywords meta tag, which told search engines what a site claimed to be about and was ignored for that reason. OpenAI and Anthropic have not documented reading the file either. Server logs shared by site owners rarely show AI crawlers fetching it.

How to create an llms.txt file

You can write it in any text editor. The spec at llmstxt.org is short, and these steps follow it.

1. 1

   Start with an H1 that names your site

   The first line is a markdown H1: a # followed by your project or company name. This is the only required part of the file. Keep it to the name alone, without a tagline.

2. 2

   Add a one paragraph summary as a blockquote

   The second element is a blockquote, a line starting with >. Write two or three plain sentences about what the site is and who it is for. Include the words a model would need to decide whether your site answers a question.

3. 3

   Add short context if it helps

   After the summary, you may add a few plain paragraphs or bullet points with details that do not fit a link list. Version notes, supported regions, or how to read the docs go here. No headings in this part.

4. 4

   Group your best pages under H2 headings

   Each H2 (##) names a group, such as Docs, Guides, or Pricing. Under it, list links as markdown bullets: [Page title](https://example.com/page): a one-line note. The note tells the model what it will find. Prefer pages that answer questions over marketing pages.

5. 5

   Put skippable links under an "Optional" heading

   The spec reserves an H2 called Optional for links a model can skip when it is short on space. Put changelogs, legal pages, and older material there. Everything above it should be the pages you would hand a new hire on day one.

6. 6

   Save it as plain text and upload to the root

   Name the file llms.txt and put it at your domain root so it is reachable at your domain root. Serve it as plain text. Check that your robots.txt does not block it and that it returns a 200 status, not a redirect.

7. 7

   Link to markdown copies if you have them

   If your platform can serve a .md version of each page, link to those instead of the HTML pages. Documentation tools do this automatically. If not, link the normal URLs. A clean HTML page is still fine.

8. 8

   Re-check it every quarter

   Dead links in the file are worse than no file. Put a quarterly reminder in your calendar to test every URL and add new key pages. Our Content Tracker can watch the file itself and alert you when it changes.

A complete llms.txt example

Here is a small example for a made-up SEO software company. Read it top to bottom and you have the whole format.

llms.txt and SEO: does it help you get cited?

No evidence says it does. Google has said it does not use the file for Search or for AI Overviews. The things that do get you cited are the same as ever. Pages that are indexable, answer the question directly, carry real facts with sources, and are mentioned by other sites. Our guide to AI search optimization walks through that work.

So where does the file fit? Treat it as hygiene, like a favicon. Add it, keep it current, and spend your real effort elsewhere. If you want to know whether AI assistants recommend you, you have to check the assistants, not the file.

Pair that with a site audit so the pages you list in the file are actually crawlable and fast. A curated list of blocked or broken pages helps nobody.

Common mistakes

The last point deserves a line. A link list without notes is a sitemap. The notes are what make the file different, because they tell the model what each page answers before it spends any context reading it.