TL;DR

• llms.txt is a proposed Markdown index file at a site’s root that gives AI tools a curated map of the site’s most important content.
• The format is a proposal, not a standard, and differs from robots.txt, sitemap.xml and llms-full.txt, which inlines the entire documentation in a larger file.
• Google has never encouraged the use of llms.txt and Ahrefs data shows that 97% of published files got zero fetches in May 2026.
• As of 2026, llms.txt is therefore useless for most marketing purposes, with the developer docs and coding agents niche being the exception. Creating an llms.txt file is not that hard: docs-heavy sites can do it in about 20 minutes at near-zero cost.
• The definitive three-step test: (1) check server logs for /llms.txt requests; (2) create the file if you find requests; (3) ignore if you don’t.

llms.txt is a proposed Markdown file that sits at a website’s root path (yourdomain.com/llms.txt) and gives AI systems a short, curated index of its most important pages. It’s not a rulebook for crawlers, but rather a reading list for language models.

But how big is llms.txt? Well, there’s a lot controversy around its importance because, even though the SEO industry pitched it as a shortcut for AI visibility, an Ahrefs llms.txt study from June 2026 shows 97% of llms.txt files never get read.

This guide simplifies llms.txt by covering what the file is, what it looks like, who uses it and whether it deserves 20 minutes of your time.

What is llms.txt?

The proposed standard llms.txt is a single Markdown index file at a site’s root that summarizes the site’s most important content and links out to it. Its main purpose is to help AI systems orient themselves without crawling every page. The llms.txt spec frames it as inference-time assistance, not crawl-time indexing.

The problem the file targets is real. Language models work with limited context windows, so they can’t ingest an entire website in one go. Raw HTML doesn’t help either, since navigation, scripts and boilerplate bury the actual content. In theory, a hand-picked Markdown index sidesteps both problems at once.

The format was first proposed by Jeremy Howard, the Answer.AI co-founder. His pitch was practical, not promotional: the idea was to give AI systems a clean map of what matters on a site and skip the noise. Ranking promises were no part of it.

Nevertheless, llms.txt is a proposal, not a ratified standard. No standards body has adopted it, nothing about it is official and no major AI platform has committed to reading it. That’s something we will have to keep in mind for the rest of the article.

What does an llms.txt file look like?

The format is deliberately minimal Markdown. The spec only has one hard requirement: an H1 with the site or project name. Everything else is optional, from the blockquote summary under the title to the H2-grouped lists of links with short descriptions.

Here’s a realistic example for a fictional SaaS docs site:

AcmeMetrics

AcmeMetrics is an analytics platform for product teams. This index lists the docs AI tools should read first.

AcmeMetrics exposes a REST API and a JavaScript SDK. Most integration questions are answered in the API reference.

Docs

• Quickstart: install, authenticate and send your first event
• API reference: every endpoint with request and response examples

Optional

• Changelog: release notes by month
• Community forum: user discussions and workarounds

1. The blockquote gives an AI a one-line summary before it reads anything else
2. Each link carries a short description, so a model can decide what to fetch without fetching it
3. The “Optional” section flags content an AI can skip when it’s short on context

Creating an llms.txt file takes minutes, not days, and you should hold on to that thought. The low cost matters later on, when the question becomes whether the file earns even that.

llms.txt vs robots.txt and sitemap.xml

Despite the filename, llms.txt is not an access-control file. It blocks and controls nothing. It’s an informational index, closer in spirit to a sitemap than to robots.txt, except it’s designed for machine reading instead of exhaustive listing.

Confusing? This is the cleanest way to hold the three apart:

• robots.txt is permissions
• sitemap.xml is inventory
• llms.txt is curation

The table further illustrates the differences:

	llms.txt	robots.txt	sitemap.xml
Purpose	Curated index of what matters, with context	Rules for what bots may crawl	Complete list of URLs
Audience	LLMs and AI agents	Search engine crawlers	Search engine crawlers
Controls access?	No	Yes	No
Format	Markdown	Plain text directives	XML
Status	Proposal (2024)	De facto standard since 1994	Adopted standard

llms.txt doesn’t replace robots.txt and sitemap.xml, coexisting with both files. Adding an llms.txt file doesn’t affect your existing crawl setup.

llms.txt vs llms-full.txt vs .md files

Readers get these files wrong all the time, so here’s a straightforward take on each of the three:

1. llms.txt: it’s the short index, comprised of a list of links with descriptions that should be small enough to read in one pass.
2. llms-full.txt: meant to inline the full text of the documentation into one large file, allowing tools to load everything at once instead of following links.
3. .md files: serve a clean Markdown copy of any page (to check the clean copy, you just have to add .md to the site’s URL).

For a better understanding of the differences between llms.txt and llms-full.txt, you just have to open your favorite website, add either llms.txt or llms-full.txt to the URL and click enter.

Anthropic is a useful example. The llms.txt version indexes everything into a short collection of essential links, whereas the llms-full.txt version is extremely long and detailed in comparison.

Which one you’d actually maintain is mostly decided for you because docs platforms generate these files automatically. That convenience leads straight into the adoption question.

Who actually uses llms.txt?

Real adoption exists. You can check it from a browser tab right now: Anthropic, Stripe, Cloudflare and Perplexity all serve live llms.txt files on their developer-docs domains. The pattern is clear: we’re looking at technical documentation, not marketing sites.

A big share of that adoption arrived automatically. Mintlify, for example, started generating /llms.txt and /llms-full.txt for every docs site it hosts, which gave thousands of sites the file at once. This is problematic because the raw adoption numbers (i.e., how many sites serve live llms.txt files) don’t necessarily tell us whether teams have intentionally decided to adopt the format or if it was automatically incorporated.

There’s one more catch: publishing an llms.tx file and having it read are different things. No major AI platform has yet committed to the format, and that begs the question:

Does llms.txt actually work?

On the evidence available in mid-2026, llms.txt does nothing for search or AI visibility on the general web. This conclusion is backed by two major sources: the 2026 llms.txt study published by Ahrefs and Google.

1 – What the Ahrefs llms.txt study shows

Remember that Ahrefs study we’ve mentioned above? This is its boldest, most enlightening take:

97% of the llms.txt files in Ahrefs’ 137,000-domain sample received zero requests in May 2026.

The small slice that did get traffic doesn’t rescue the story either, since only 19.5% of those fetches came from AI tools. One caveat cuts both ways: this measures fetch traffic to the file itself, not AI citation outcomes. However, what’s the point of llms.txt files if nobody reads them?

2 – What Google says

Google’s initial position was loud and clear, as reported by Search Engine Land in July 2025: llms.txt was not to be used for SEO and the AI Overviews. That position, however, has recently changed. In mid-2026, Google stated that it’s fine to use llms.txt for AI SEO despite being previously dismissive of the format, to the point of discouraging its use.

Google’s change of heart is encouraging for llms.txt adopters, but it doesn’t change the fundamental fact that the vast majority of files aren’t read. This helps to explain why Google’s John Mueller compared llms.txt files to the old keywords meta tag. At least for now, both are existing signals that search systems simply don’t consume.

Nevertheless, the importance of the llms.txt format is still open for debate…

The counter-case: docs and coding agents

The data isn’t uniformly negative. Within the small share of files that actually got fetched, GPTBot led and Claude-Code came second, both ahead of all the search and assistant bots. That’s coding tools reading developer docs, not search engines evaluating marketing sites.

Mueller’s own framing points in the same direction. He called llms.txt a “temporary crutch” that might save AI coding tools some tokens while they parse developer docs. Some individual site owners report heavier traffic, like one operator whose logs showed OpenAI hitting his llms.txt every 15 minutes. That’s one site’s logs, not a trend.

The honest read? llms.txt files are not a ranking leveler for everyone, but they can be a real niche for docs and agents.

Common misconceptions about llms.txt

If you’re still a bit confused, there’s a good chance you’re being misguided by one of these four llms.txt misconceptions:

1. “llms.txt boosts your rankings or AI visibility.” The spec never claimed this. Not once. This myth is a sole product of the SEO industry.
2. “Google told SEOs to create it.” Not true. Google’s initial position was actually the opposite, and despite a recent change of heart, the company never endorsed the format.
3. “It controls AI crawlers like robots.txt.” llms.txt files are an index, not a directive. Unlike robots.txt, they block nothing.
4. “99.14% of llms.txt files got zero requests.” This is an exaggerated and unverified stat published in some sources. The official number is 97%, as reported by the Ahrefs study.

Should you create an llms.txt file?

Yes, the importance of llms.txt files has been grossly overstated by the SEO industry, but that doesn’t mean the answer is always no. In sum:

1. Creating an llms.txt file is worth it if you run developer docs or API reference content. Coding agents already fetch these files, the cost is near zero and there’s no downside. It takes 20 minutes or less, so why not?
2. It’s not worth it if you’re a general marketing site hoping for SEO or AI-visibility gains. We wish answer engine optimization was this easy, but the evidence says the bots aren’t reading llms.txt files. In this case, even 20 minutes feels like a waste…

If you’re in the worth-it camp, here’s how easy it is to create an llms.txt file:

1. Decide which pages an AI should read first (usually, your quickstart and reference docs);
2. Write the Markdown file: one H1, a summary blockquote and grouped link lists;
3. Upload it to your site’s root so it resolves at yourdomain.com/llms.txt.

An important detail to keep in mind: if your docs live on GitBook or Mintlify, you probably already have an llms.txt. These files can easily go unnoticed, so don’t forget to check before you build one by hand.

The definitive llms.txt test

In conclusion, llms.txt is a real, simple, live proposal whose proven value is extremely narrow. As of mid-2026, it’s a useful low-cost index for docs sites and coding agents. If you don’t fit the bill, there’s no real reason to worry about the format.

But hey, isn’t utterly ignoring llms.txt ill-advised? Well, there’s one easy test that settles the matter:

Before spending anything on llms.txt, check your server logs for /llms.txt requests. If AI bots are already asking for the file, the 20-minute setup is worth it. If not, place your time and energy elsewhere.

To learn more about how AI systems talk about you right now, you just have to run your brand through CrowdReply to see how the engines actually describe you.

FAQ about llms.txt files:

Do AI crawlers actually read llms.txt?

Mostly, they don’t. 97% of published llms.txt files got zero requests in mid-2026 and Google initially stated its systems don’t use the file at all. The exception is developer documentation, where coding-agent bots like GPTBot and Claude-Code do fetch the files.

Does llms.txt help SEO?

No. Google has said on the record that it does not use llms.txt and that appearing in AI Overviews takes normal SEO, nothing extra. John Mueller compared it to the old keywords meta tag, a signal search systems simply don’t consume.

Is there an official llms.txt standard?

No. llms.txt is a proposal, not a ratified standard. No standards body has adopted it and no major AI platform has committed to reading it. For now, it remains community-driven.

Should I create an llms.txt file?

You should do it for docs-heavy or API-reference sites, where the cost is near zero and coding agents already fetch the files. It’s not worth it for general marketing sites expecting visibility gains.

What is llms-full.txt?

A companion file that inlines the full text of a site’s documentation into one large file, where llms.txt itself is only a short index of links.

Where should llms.txt live?

llms.txt should live at the site’s root path, as in yourdomain.com/llms.txt (the same convention robots.txt uses). Subdomain roots work the same way.