Does llms.txt work? What the proposal says and what search and AI companies document

The short answer. For Google Search, no: Google says Search does not use llms.txt and that you do not need it. For ChatGPT, Claude and Perplexity, nobody can say from the documentation, because the crawler documentation from OpenAI, Anthropic and Perplexity does not mention the file. It is a proposal, not a standard. Publishing one is cheap and does no harm if it is generated from the same data as your pages; it is not a substitute for crawler access, clear pages or structured data.

What is llms.txt?

A proposal for a Markdown file at /llms.txt that summarises a site for language models and links its most useful pages. The proposal, first published by Jeremy Howard in September 2024, describes it as “A proposal to standardise on using an /llms.txt file to provide information to help agents use a website.”

The format is small:

• An H1 with the name of the site. The proposal says “This is the only required section.”
• A blockquote with a short summary.
• Optional paragraphs or lists with more detail, without headings.
• H2 sections of links, each a Markdown link with an optional note after a colon. A section named “Optional” holds links an agent can skip.

The proposal also asks sites to offer a Markdown version of each page at the same URL with .md added. It positions the file next to the existing ones, not in place of them: it says llms.txt “can complement robots.txt by providing context for allowed content”, and that a sitemap “isn’t a substitute for llms.txt”.

What does Google say about it?

Google answers the question directly. Its guide to generative AI features in Search says: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search”, because Google Search itself does not use them. The same page lists “creating unnecessary AI text files (like llms.txt)” among the things to skip.

Google is also clear that publishing one is not a problem. The guide says it is “completely fine” to create and maintain llms.txt files for other services that use them, and that this has no effect on Search. The AI features page adds that there are “no additional requirements” to appear in AI Overviews or AI Mode.

So for Google, the file is neutral: no benefit, no penalty.

What do OpenAI, Anthropic and Perplexity document?

Nothing about llms.txt. Each company’s crawler page describes its agents, their user-agent strings, what they are used for and how they treat robots.txt:

• OpenAI’s crawler overview covers GPTBot, OAI-SearchBot and ChatGPT-User.
• Anthropic’s support article covers ClaudeBot, Claude-SearchBot and Claude-User.
• Perplexity’s crawler page covers PerplexityBot and Perplexity-User.

None of them lists llms.txt as something the crawlers read, and none asks site owners to publish one. The controls they do document are robots.txt rules and, for Perplexity, firewall allowlisting by IP range.

One point causes confusion. The proposal notes that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer documentation. That shows the companies use the format to describe their own docs; it says nothing about whether their crawlers read yours.

• The proposal

  • Defines the format
  • Complements robots.txt and sitemaps
  • Open for community input

  A proposal, not a standard

• Google

  • Search does not use it
  • Not needed for AI Overviews or AI Mode
  • Fine to publish for other systems

  Neutral for Search

• OpenAI, Anthropic, Perplexity

  • Crawler docs do not mention it
  • robots.txt is the documented control

  Undocumented either way

Should you publish one?

Decide on cost, not on expected visibility.

Reasons to publish one. It is a short public statement of what the site is and which pages matter. The proposal says thousands of sites publish one and names platforms that generate it, including Mintlify, GitBook and Yoast SEO. If it is generated at build time from the same data as the pages, it cannot drift out of date.

Reasons not to bother. No search or AI company documents reading it, so there is no measurable benefit to point to. A hand-maintained copy goes stale and can contradict the site, which is worse than having none.

What it does not replace. Crawler access in robots.txt and at the firewall, content in the HTML response, and structured data that matches the page. Those are documented inputs. Fix them first.

How do you check whether anything reads it?

From your server logs. Filter requests for /llms.txt and /llms-full.txt, and group them by user agent. Verify any agent that claims to be a known crawler against its operator’s published IP ranges, because a user-agent string can be forged; our AI crawler reference links each operator’s verification method. Requests tell you something fetched the file, not that it was used in an answer.

What we don’t claim

• We do not claim llms.txt improves rankings, mentions or citations. No vendor documentation linked here says it does, and Google says Search does not use it.
• We do not claim no system reads it. The documentation is silent, which is not the same as a no.
• We do not claim the details are permanent. The proposal and the vendor pages change; re-read them before relying on this.