What Is llms.txt? The Proposed File for AI, Explained
Last updated:
llms.txt is a Markdown file at the root of a website, such as https://example.com/llms.txt, that gives large language models and the agents built on them a short map of the site: its name, a one-line summary, and lists of links to its most useful pages, each with a note. It's a proposal, not a standard, and whether AI assistants read it is mostly unconfirmed. This guide covers where llms.txt came from, its format, what it can and can't do, and how to add one if you decide to.
Where llms.txt came from
Jeremy Howard proposed llms.txt on September 3, 2024, at llmstxt.org. The problem it addresses is practical: a web page made for people is full of navigation, scripts and layout, and a language model with a limited context window does better with a clean, short text. The proposal suggests two things. First, a file at a predictable address that says what the site is and links to the pages worth reading. Second, a clean Markdown copy of each of those pages, at the page's own address with .md added.
Its model is robots.txt and the sitemap, two files at predictable addresses that crawlers already know to look for. But where those two are read by every major search engine, llms.txt asks AI tools to adopt a new habit.
The llms.txt format
The file is Markdown with a fixed order of parts:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary: what an agent needs to know to make sense of the rest.
- Any other Markdown except headings: paragraphs or lists with more detail about the site and how to read the links.
- H2 sections, each a list of links written
[name](url), optionally followed by a colon and notes about the page. - An "Optional" section, by convention: links an agent can skip when it needs a shorter context.
A small shop's file could look like this:
# Example Shop
> Trail running shoes and gear, sold online and shipped from Denver, Colorado.
We answer email within one business day. Prices are in US dollars.
## Products
- [Trail running shoes](https://www.example-shop.com/shoes/trail/): Every model, with sizes and weights
- [Size guide](https://www.example-shop.com/help/sizes/): How our sizes compare with other brands
## Help
- [Shipping and returns](https://www.example-shop.com/help/shipping/): Delivery times and costs; returns are free within 60 days
## Optional
- [Blog](https://www.example-shop.com/blog/): Running guides and product news
The file can also sit at a path, such as /docs/llms.txt, covering the pages under it; the proposal says an agent should use the most specific file that applies. Check any file against the format with the llms.txt validator, which lists problems by line.
Who supports llms.txt
This is the part to be honest about. As of November 2026 we haven't found documentation from OpenAI, Anthropic, Google, Microsoft or Perplexity saying that their crawlers or assistants read llms.txt when they answer questions. Google goes further for its own AI features: its documentation says you don't need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode, and that a page needs to be indexed and eligible to be shown with a snippet.
Its clearest use today is by hand: a person can give an AI assistant a project's llms.txt, or the pages it links to, as context for a question. Schema.biz publishes one for that reason, at /llms.txt, generated from the same list of tools as the site's menus.
So treat it as a cheap, optional step. It costs a few minutes and does no harm, but it isn't a ranking factor, and no file can promise that an AI assistant will cite you.
llms.txt, robots.txt and sitemaps compared
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| What it says | Which crawler may fetch which paths | Which pages exist, and when they changed | What the site is, and which pages matter most |
| Written for | Crawlers | Search engines | Language models and agents |
| Status | Standard (RFC 9309) | Protocol read by the major search engines | Proposal; support unconfirmed |
| Controls access? | Yes, for crawlers that obey it | No | No |
| Check it with | Robots.txt checker | Sitemap checker | llms.txt validator |
The first two matter more. If robots.txt blocks AI search crawlers such as OAI-SearchBot or PerplexityBot, an llms.txt won't let them in; if a page is noindex, Google won't show it in AI Overviews. The AI readiness checker tests a page for all of these at once.
How to create an llms.txt
- Draft it. The llms.txt generator reads your sitemap and pages and drafts the file, grouped into sections, with each page's title and description. Or write it by hand: it's plain Markdown.
- Edit it. Keep the pages that answer the questions people ask about you: what you sell or do, prices, policies, documentation, contact. Write notes that say what's on each page. Put background reading under "Optional".
- Use full URLs. An agent may read the file away from your site, so
https://example.com/docs/, not/docs/. - Publish it at the root, so it answers at
/llms.txt, served astext/plainortext/markdown: usually the folder that holds robots.txt. - Check the live file. Run the validator on your address. A host that answers unknown addresses with the home page serves HTML at
/llms.txtuntil the file is really there. - Keep it current. A link to a page that moved is worse than no link. Update the file when your main pages change.
Frequently asked questions
FAQ
Is llms.txt an official standard?
No. It's a proposal published at llmstxt.org in September 2024, not a standard from a body such as the IETF or W3C, and support from the major AI providers is limited or unconfirmed. Robots.txt, by contrast, is a standard: RFC 9309.
Does Google use llms.txt?
Google's documentation on AI features says you don't need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode. A page needs to be indexed and eligible to be shown with a snippet. llms.txt doesn't change either.
Can llms.txt block AI crawlers or AI training?
No. It's a list of links, not a set of rules. To control which crawlers may fetch your pages, use robots.txt and each crawler's own token; the robots.txt checker shows what a file blocks.
What's the difference between llms.txt and llms-full.txt?
llms.txt is the proposal's file: a short index of links. llms-full.txt is a practice some documentation sites follow, putting the full text of their docs in one file; the proposal at llmstxt.org doesn't define it.
How long should an llms.txt be?
The format sets no limit. It's meant as a short, curated map, so list the pages that answer the questions people ask about you, each with a note, and put background material under "Optional". If it starts to look like your sitemap, it's too long.