What llms.txt is, and which AI engines have confirmed reading it

What is llms.txt?

Jeremy Howard proposed llms.txt as a Markdown file placed at /llms.txt on a website, listing the pages and clean Markdown documents an AI agent should read first when it needs to understand the site. The proposal carries the dateline 3 September 2024 on llmstxt.org, and the text published there now is labelled v2. The file curates instead of enumerating: it names a handful of high value URLs and describes each one, where a sitemap lists everything indexable.

The specification states the motivation in one sentence: "Agents are best served by concise, expert-level information gathered in a single, accessible location." Nothing in the proposal grants access, withholds access, or obliges any engine to fetch the file. It is a suggestion published at a predictable path.

Last verified: 9 September 2026.

What does an llms.txt file have to contain?

An H1 is the only element the specification requires, and every other section is optional. The spec fixes the order, which is what lets a parser or a regex read the file without a language model.

The shape the spec publishes as its own mock example:

# Title
 
> Optional description goes here
 
Optional details go here
 
## Section name
 
- [Link title](https://link_url): Optional link details
 
## Optional
 
- [Link title](https://link_url)

Two placement rules follow. The file sits at the site root or at any subpath, and "a file covers the URLs under its path, and where more than one file applies, agents should use the most specific one," so /docs/llms.txt covers /docs/ only. The same proposal asks sites to serve a clean Markdown twin of each page at page.md or page.html.md, and to advertise both with rel="alternate" type="text/markdown" and rel="describedby" link relations, in HTML or in an HTTP Link: header.

Which AI engines have confirmed they read llms.txt?

No engine operator states in its crawler documentation that its products read llms.txt, and the three largest vendors all publish one for their own developer docs. Publishing a file and consuming a file are separate acts, and the gap between them is the whole story of this format.

The nearest thing to vendor acknowledgement comes from Chrome. Lighthouse ships an llms.txt audit in its agentic browsing checks, describing the file as "an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents," and warning that "without this file, agents may spend more time crawling the site to understand its high-level structure and primary content." That audit names no engine that reads one. Treat every adoption number you see for llms.txt with suspicion unless it names a source and a date. The spec's own claim is qualitative: "thousands of sites publish an llms.txt file."

How is llms.txt different from robots.txt?

robots.txt is a fetch-permission file with an RFC behind it and documented support from every major crawler operator, and llms.txt is a content-curation proposal with no enforcement and no confirmed consumer. The specification draws the same line itself: "robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user."

Both files are public and neither is a legal control. Which crawler actually arrives is a question server logs answer, not either file, and the entries for CCBot and ChatGPT-User cover what each user agent means when it shows up.

What is llms-full.txt?

Mintlify originated llms-full.txt as a companion file that, in its words, "combines your entire documentation site into a single file as context for AI tools and LLM indexing," with each page rendered as its title, source URL, description and full Markdown body. The llms.txt specification does not define llms-full.txt, does not mention it, and does not require it. Documentation platforms generate the two together, which is why the pair appears on so many docs sites at once.

The practical split: llms.txt is an index small enough to sit in a context window, and llms-full.txt is the corpus behind it. A site that publishes only the full dump has handed an agent a file too large for the purpose the index was designed to serve.

Who should publish an llms.txt file?

Documentation teams get the clearest return, because coding agents are the one audience observed following the format. The specification says so directly: "llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials." A coding agent pointed at your API docs fetches the URLs you name, which is a different situation from a search engine deciding what to cite.

For a marketing site, publishing costs an hour and promises nothing measurable today. Write it if your content is genuinely hard to parse from HTML, and do not write it instead of the work that has evidence behind it. Getting cited by an assistant still runs through crawlable pages, clear answers and third-party mentions, which is the territory generative engine optimization and answer engine optimization cover.

How do you check whether anything reads your llms.txt?

Grep your access logs for requests to /llms.txt and read the user agent on each hit. A request from an agent user agent means something fetched the file, an empty result over a month means nothing did, and either answer costs you one command. That log line is the only first-party evidence available, since no vendor reports llms.txt fetches in a console.

Fetch counts still do not tell you whether an assistant quoted you. For that, watch the answers themselves: RedReplier tracks the public conversations naming your brand on Reddit, X, Bluesky, Facebook and Hacker News, which is where assistants go looking when they answer questions in your category.

Frequently Asked Questions

Does llms.txt help SEO?

No documented ranking effect exists. Google's AI features documentation does not mention llms.txt and names robots.txt as the control for how sites are crawled for Search. Publishing the file changes nothing about how Googlebot fetches or ranks your pages, and no Google documentation ties the two together.

Where does llms.txt go on a site?

At /llms.txt on the site root, or at any subpath such as /docs/llms.txt. The specification says a file "covers the URLs under its path, and where more than one file applies, agents should use the most specific one." A subpath file is the right choice when only part of your site is worth guiding an agent through.

Is llms.txt an official standard?

No. llms.txt is a proposal published by Jeremy Howard at llmstxt.org, hosted in the AnswerDotAI/llms-txt repository and open to community input. It has no RFC, no standards body and no compliance requirement, which is the difference between it and RFC 9309, the document behind robots.txt.

What is the difference between llms.txt and sitemap.xml?

A sitemap lists every indexable page for search engines, and llms.txt lists a curated few for agents. The specification argues a sitemap is no substitute because it omits Markdown versions of pages, excludes helpful external URLs, and covers documents that in aggregate exceed a model's context window.

Do I need llms-full.txt as well?

No. The llms.txt specification does not define llms-full.txt at all. The convention comes from Mintlify, which generates both for the docs sites it hosts, and the full file is a single-file dump of an entire documentation set, not an index.

Should I block AI crawlers instead of guiding them?

That is a robots.txt decision, and it turns on whether citations matter more to you than training tokens. Anthropic, OpenAI and Google each run separate user agents for training and for search, so blocking one does not have to block the other. Publishing an llms.txt neither replaces nor overrides any of those rules.

Who created llms.txt?

Jeremy Howard proposed the format, carrying a dateline of 3 September 2024 on llmstxt.org. The text published there today is labelled v2, and the project lives in the AnswerDotAI/llms-txt repository, open to community input. No standards body sits behind it.

Does ChatGPT read llms.txt files?

OpenAI publishes its own llms.txt at developers.openai.com/llms.txt for its developer docs, but nothing in its crawler documentation states that ChatGPT or any OpenAI product reads one. That documentation covers robots.txt, user agents and IP ranges, and links the company's own llms.txt only as an index to its docs. Publishing the file on your own site tells you nothing about whether OpenAI's crawlers consume it elsewhere.

What happens if you don't publish an llms.txt file?

Nothing breaks. The specification draws the contrast with robots.txt itself: ignoring robots.txt breaks a long-standing crawling convention, while ignoring llms.txt has no documented effect. Chrome's Lighthouse audit warns that without the file, agents may spend more time crawling a site to find its structure, but that is a cost in agent effort, not a penalty on the site.

Do coding agents actually use llms.txt?

The specification says llms.txt files "are used most heavily for software documentation, where coding agents follow them to find API references and tutorials," the one audience the format has documented evidence for. That is a different situation from a search engine deciding on its own what to cite. A coding agent pointed at your API docs fetches the exact URLs you name there.