llms.txt: what the file does and who reads it

// In this article+

Most llms.txt files are never requested. In May 2026, 97% of the files Ahrefs tracked got zero fetches. The file is a proposal: a short markdown page at the root of a site that lists your best pages for an AI agent to read on demand. It does not control crawling and it does not affect rankings.

That makes it a documentation convenience. Whether it earns an hour of your time depends on who reads it, and your server logs can tell you.

What is llms.txt and what does the file do?

The proposal comes from Jeremy Howard. On the llmstxt.org page ↗︎, he proposes adding a "markdown file to websites to provide LLM-friendly content." The file sits at /llms.txt, or at any path to cover the pages under it.

The format is strict and short. An H1 with the site name is the only required part. After it come an optional blockquote summary, optional notes, and H2 sections holding lists of links, each with a short note.

# Example Co
> Billing API and dashboard for small teams.

## Docs
- [Quickstart](https://example.com/docs/quickstart.md): first API call in five minutes
- [API reference](https://example.com/docs/api.md): every endpoint and error code

## Optional
- [Changelog](https://example.com/changelog.md)

An agent reads the small file, picks a link, and fetches only that page. The detail lives behind the links.

llms.txt vs robots.txt vs sitemap.xml

Three root files, three jobs. The IETF's robots.txt standard ↗︎ defines the rules as a way to "control how content served by their services may be accessed, if at all, by automatic clients known as crawlers." Access control is the whole point.

A sitemap is a different thing. The sitemaps.org protocol ↗︎ says it "enables you to provide details about your pages to search engines." It lists every indexable URL for a search engine.

llms.txt blocks nothing. A rule in it cannot keep a bot out. If the goal is access control, the file to edit is robots.txt.

Does Google or any AI search engine use llms.txt?

Google Search does not. Its guide to generative AI features ↗︎ lists llms.txt under things you can ignore. It says creating the file "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

A note on terms. GEO (generative engine optimization), AEO (answer engine optimization) and AI search optimization name the same discipline: earning visibility in AI answers. This article says GEO from here on. Google's guide treats GEO as SEO under another name. The wider argument is in GEO vs SEO: what changes when the engine answers.

Why Lighthouse audits it anyway

Chrome's Lighthouse has an experimental agentic browsing category. One audit in it, per Chrome's Lighthouse documentation ↗︎, "Checks for the presence of a machine-readable summary at the domain root." Search Engine Journal explained the apparent contradiction ↗︎: "llms.txt isn't needed for AI features, while Lighthouse now checks the file for agentic browsing readiness in an experimental audit."

Two products, two questions. Search asks whether to rank you. The agentic audit asks whether a browsing agent can orient itself. Ignore the Lighthouse result for Search purposes.

What OpenAI and Perplexity document instead

Neither vendor's crawler page tells site owners to publish llms.txt. They document bot names and robots.txt controls. OpenAI's crawler overview ↗︎ says "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." Whether ChatGPT search can show your pages is a robots.txt decision.

Perplexity's crawler page ↗︎ asks site owners to allow PerplexityBot in robots.txt. It also names Perplexity-User and notes that "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

Checklist for the sections above:

• Google Search ignores llms.txt, per Google.
• Lighthouse checks for it in an experimental category.
• OpenAI and Perplexity document robots.txt and named bots, not llms.txt.

Who actually requests llms.txt?

Mostly nobody, and mostly not AI search. The best evidence is the Ahrefs study ↗︎ of server logs and live traffic from 137,210 domains that received traffic in May 2026.

The headline numbers:

• 28% of the domains publish an llms.txt file, about 38,000 domains.
• 97% of those files received zero requests in May 2026.
• Of the files that were requested, 96% of requests came from bots.
• Named AI bots made 19.5% of the requests that did arrive.
• AI retrieval bots, such as OAI-SearchBot and PerplexityBot, made 1.1%.

The 1.1% is the number that matters for GEO. The bots that fetch pages to answer live queries barely touch the file. Ahrefs found GPTBot, a training crawler, was the top AI fetcher, with Claude-Code, Anthropic's coding agent, second.

SEO audit tools made 21.7% of requests, ahead of any AI category. Tools that study llms.txt itself, such as GEO checkers and validators, made 12%. Ahrefs also found zero AI bot requests for llms.txt files that do not exist. AI bots do not go looking.

Read these as ceilings. Ahrefs says "fetched" does not mean "read." The sample also skews technical, so Ahrefs treats the 28% adoption figure as an upper bound. It is one vendor's analytics population, not a census of the web.

Adoption is growing, requests are not

Search Engine Journal's coverage of the Ahrefs data ↗︎ quotes John Mueller, who called the file "not done for search" and "a temporary crutch, perhaps to save some tokens" for AI coding tools. PPC Land's report on adoption ↗︎ says "the number of sites implementing these files has grown sharply." It cites Originality.ai tracking of 3 million sites: 4,088 llms.txt instances in June 2025, 36,120 by May 2026.

Adoption grew 8.8x. Use did not follow. Platforms generating the file by default account for part of that gap.

How do I check my own server logs for llms.txt requests?

Do this before you publish anything. The procedure below is a method, not something taken from a source. It needs 30 days of access logs from your server or CDN.

Filter by path and user agent

On a standard combined-format access log, pull every request for the file and count user agents:

zgrep -h "GET /llms.txt" access.log* \
  | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn | head -20

If you use a host without raw logs, use your CDN's bot report. The Ahrefs study ↗︎ describes a similar filter in its Bot Analytics: page URL contains llms.txt. Also record the status code. A 404 means bots asked for a file you never published.

Then sort each user agent against a lookup. The categories below come from the Ahrefs study.

Source: Ahrefs, llms.txt study ↗︎

Read the result

Count the rows that are retrieval bots, assistants or agents. Ignore audit tools, profilers, link-preview bots and your own checks. A hit from a scanner is not AI use.

Three outcomes. Zero requests: the file has no readers, which is the base rate. Only audit tools and unknown bots: same conclusion. Coding agents or assistants present: the file has an audience, and it is a developer one. Even then, a fetch is not proof that anything acted on the content.

For wider measurement of where AI answers mention you, see AI visibility. Logs only show who fetched a file, not who cited you.

When is llms.txt worth an hour?

Decide on the log result and your audience, not on the trend. Google says the file is harmless to Search. The Ahrefs study concluded that "the cons outweigh the pros right now," and called a published file largely decoration for ChatGPT, Perplexity or AI Overview visibility.

Checklist, in order:

1. Do coding agents or docs tools read your site? Publish the file. This is where Ahrefs found the closest thing to an intended audience.
2. Do you ship API or developer documentation? Publish it, and generate it from your docs platform if one offers it.
3. Is the site a marketing site chasing citations? Skip it for now. Spend the hour elsewhere.
4. Did your logs show retrieval bots fetching a file you already have? Keep it current and link to it from your docs.

Developer docs and API products

Docs are the use case in the proposal itself. Coding agents fetch an API reference to get a call right. A generated file with links to markdown pages costs little. Keep it small, link only to pages you control, and treat it like code. Ahrefs flags prompt injection as a risk, because agents trust what the file says.

Marketing sites

Google's guide says the basics decide visibility: pages must be indexed and snippet-eligible, and content must be crawlable. For more on how models choose what to cite, read LLM SEO. The work that moves GEO sits upstream of this file, in agentic SEO.

What to do this week

1. Run the 30-day log check above. Group by user agent and note the status code.
2. Open robots.txt. Confirm it allows OAI-SearchBot and PerplexityBot if you want ChatGPT search and Perplexity to surface you. The file is the control, not llms.txt.
3. Confirm your key pages are indexed and snippet-eligible in Google Search Console.
4. If you ship docs, publish a short llms.txt, generated by your platform where possible. Link it from your docs.
5. Ignore the Lighthouse llms.txt result for Search purposes. Re-run the log check in 90 days.

FAQ

Does llms.txt improve Google rankings?

No. Google's guide to generative AI features says creating the file will neither harm nor help your visibility or rankings in Google Search, because Google Search ignores it. That covers AI Overviews and AI Mode too.

Is llms.txt the same as robots.txt?

No. The IETF's robots.txt standard defines an access-control file for crawlers. The llms.txt proposal describes a curated markdown overview that an agent uses on demand. The proposal says robots.txt and llms.txt have different purposes.

Should I block AI crawlers with llms.txt?

No. The file cannot block anything. Use robots.txt, and use the bot names the vendors publish. OpenAI documents OAI-SearchBot, and Perplexity documents PerplexityBot and Perplexity-User. Perplexity notes that its user-triggered fetcher generally ignores robots.txt.