llms.txt Explained: Should You Add It?

The distinction matters: robots.txt is about permission, sitemap.xml is about discovery, and llms.txt is about comprehension. It doesn't grant or deny access, and it isn't meant to list every URL - it's an editorial shortlist.

What the file looks like

The spec defines a light, predictable markdown structure that's easy to write by hand. It opens with an H1 site name, an optional blockquote summary, then H2 sections containing lists of links with short descriptions:

# Your Company

> A one-line summary of what your site or product does.

## Docs

- [Getting started](https://example.com/docs/start): Install and first steps

- [API reference](https://example.com/docs/api): Endpoints and auth

Guides

• Pricing: Plans and limits

There's also a companion convention, **llms-full.txt**, which inlines the actual content of those pages into one long markdown file - useful when a tool wants the full text in a single fetch rather than following links.

## The Problem It Tries to Solve

**llms.txt exists because HTML is a bad delivery format for machines that read in tokens, not pixels.** When an AI system tries to understand a page, it has to strip away layout, scripts, and clutter to find the substance - and it often does that imperfectly, especially on JavaScript-heavy sites.
A curated markdown digest sidesteps that. It says, in effect: don't guess what matters on my site - here's the shortlist, already cleaned up. For documentation-heavy products, developer tools, and knowledge bases, that's genuinely valuable, particularly for retrieval-augmented generation (RAG) systems and AI coding assistants that you or your partners point directly at the file.
That last point is the honest core of llms.txt's value today: it is most clearly useful in contexts where *someone deliberately chooses to consume it* - an internal RAG pipeline, an agent you built, a docs platform that ingests it. It is far less proven as a passive signal that public AI engines pick up on their own.

## Who Actually Supports It (Be Honest)
![What llms.txt can and cannot do for AI visibility](https://sv8eruuuunmdjop0.public.blob.vercel-storage.com/blog/llms-txt-explained/llms-txt-reality-check-v1.webp)

**llms.txt has moved from a niche proposal to an emerging convention, but that is not the same as confirmed retrieval by the major answer engines.** The [original specification](https://llmstxt.org/) still describes itself as a proposal for helping LLMs use a site at inference time.
Publisher-side support is meaningful:

- Documentation platforms and developer tools can generate `llms.txt` and `llms-full.txt` automatically.

- In May 2026, [Chrome Lighthouse added an llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt). Lighthouse calls the file an emerging convention and checks that a provided file does not return a server error. A missing file returns “Not Applicable,” because the file remains optional.

Consumer-side evidence is still limited. OpenAI, Anthropic, and Perplexity have not publicly committed to using `llms.txt` as a citation or retrieval signal. Google is explicit: its [guide to generative AI features in Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says Google Search does not use `llms.txt`, and that maintaining the file neither helps nor harms visibility or rankings in Search, AI Overviews, or AI Mode.
The fair summary: llms.txt now has credible tooling support and a clearer agent-discovery use case, but no evidence that it improves organic visibility in the major answer engines.

## Does It Help AI Visibility? The Honest Answer

**There is still no public evidence that adding llms.txt improves citations or visibility in ChatGPT, Claude, Perplexity, Gemini, Google AI Overviews, or AI Mode.** Lighthouse support validates the file as an optional agentic-browsing convention; it does not turn it into a search ranking signal.
That does not make the file worthless. Its value is narrower and easier to defend:

- **Where it can help:** agents, coding tools, RAG systems, internal assistants, and partners that deliberately look for the file.

- **Where it remains unproven:** organic AI search visibility and citation selection by major consumer answer engines.

- **Where it helps operationally:** giving teams a curated, versionable map of the pages they most want machines to understand.

The cost is low, so a clean implementation can be a reasonable experiment. Just label the experiment honestly and measure crawler requests and citation outcomes instead of promising a ranking gain.

## How to Create an llms.txt (Step by Step)

**If you decide to add one, do it properly and keep it current - a stale llms.txt is worse than none, because it actively misdirects.** Here's a clean, low-effort process.
1. **Decide if you even have curatable content.** llms.txt shines for docs, guides, references, and knowledge bases. If your site is a five-page brochure, the payoff is minimal.
2. **Shortlist your most important URLs.** Think editorially: the pages you'd want an assistant to read first to represent you accurately. Quality over completeness.
3. **Write the file in the spec format.** Start with an H1 name, add a one-line blockquote summary, then group links under H2 sections with a short description after each link (see the example above).
4. **Optionally generate llms-full.txt.** If a consuming tool prefers the full text in one fetch, produce a companion file that inlines the page contents. Many docs platforms can do this automatically.
5. **Host it at the root.** Serve the file at `https://yourdomain.com/llms.txt` as plain text/markdown, publicly accessible, no auth wall.
6. **Keep it in sync.** Add it to your release or publishing checklist so it updates when key pages change. Automate generation if your platform supports it.

Quick checklist

• File served at /llms.txt, publicly reachable

• H1 name + one-line summary present

• Only your highest-value pages listed, each with a short description

• Links are absolute, correct, and not broken

• Optional llms-full.txt for full-text consumers

• An owner and a cadence to keep it fresh

The only honest llms.txt experiment

If you implement llms.txt, test it like a product change instead of declaring success when the file returns 200.

1. State the hypothesis. Example: “Publishing a curated llms.txt will increase machine retrieval of our documentation entry points.” Do not use “improve AI rankings,” because no major engine has confirmed that mechanism.
2. Capture a baseline. Record requests to /llms.txt, crawler visits to the linked pages, AI citations, and cited URLs before the file exists.
3. Choose a treatment set. Link a defined group of important pages in llms.txt. Keep a comparable group outside the file as a directional control.
4. Hold other variables steady. Avoid rewriting titles, changing internal links, and launching digital PR for the treatment pages during the same observation window.
5. Measure the chain, not one metric. A request to llms.txt shows discovery. More crawling shows use. A citation change shows an outcome. These are three different events.
6. Predefine the decision. Keep the file if it improves agent workflows, simplifies machine navigation, or produces a repeatable retrieval signal. Maintain it as low-cost infrastructure if neutral. Remove or deprioritize it if it creates operational work without observable use.

The experiment cannot prove causation in every external engine, but it prevents the most common error: treating adoption of a file as evidence of visibility impact.

Realistic Expectations vs the Hype

Treat llms.txt as a cheap, low-risk experiment - not a growth strategy. The durable levers of AI visibility are unglamorous and well understood: be crawlable, structure your content so answers are easy to extract, build genuine authority and entity clarity, keep your material fresh, and earn citations on third-party sources. A markdown file at your root changes none of those fundamentals. The most important discipline here is measurement. Because the AI providers haven't confirmed whether they use llms.txt, you shouldn't take anyone's word - including this article's - for whether it moves the needle for your site. Watch what actually happens: are AI crawlers requesting the file, and is your citation share changing after you ship it? That's exactly the kind of before/after signal a monitoring platform is for. Qwairy tracks how your brand shows up across ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews - including which sources get cited and how AI crawlers behave on your site - so you can test a change like adding llms.txt against real visibility and citation data instead of hoping. If it correlates with more citations, great. If it doesn't, you've spent an afternoon and learned something concrete.

Prioritize proven foundations first: Run the technical GEO checklist, make key pages discoverable through site architecture for AI, and configure AI crawler access before treating llms.txt as an experiment.

Conclusion

llms.txt is a smart, well-intentioned proposal for a problem that's real: AI systems struggle to read cluttered HTML, and a curated markdown guide is an elegant fix. But the standard only works if the consumers adopt it, and today the major AI engines haven't confirmed they do. Add it if it's cheap for you - especially if you run documentation and your platform generates it automatically. Keep it accurate. And don't confuse shipping a file with earning visibility. The things that actually get you cited by AI are the same fundamentals as ever; llms.txt is, at best, a small optional extra on top. Measure, don't assume.

FAQ

Is llms.txt an official standard?

No. It's a community proposal published at llmstxt.org, introduced by Jeremy Howard in September 2024. It has a clear specification and a growing base of sites that implement it, but it is not ratified by any standards body and not endorsed as an official protocol by the major AI providers.

Do ChatGPT, Perplexity, or Google use llms.txt?

Google explicitly says Search does not use llms.txt. OpenAI, Anthropic, and Perplexity have not publicly confirmed that their systems use it for retrieval, grounding, or training. Some independent analyses suggest major AI crawlers rarely request the file at all. Treat any visibility benefit outside Google as unproven and measure it as an experiment.

What's the difference between llms.txt and robots.txt?

They solve different problems. robots.txt controls permission - which crawlers may access which paths - and is widely honored. llms.txt is about comprehension - a curated, human-picked guide to your best content for language models. One gatekeeps access; the other tries to aid understanding.

What is llms-full.txt?

It's a companion file that inlines the full text of your key pages into a single markdown document, rather than just linking to them. It's handy when a consuming tool wants everything in one fetch. Many documentation platforms can generate it alongside llms.txt automatically.

Will llms.txt improve my SEO or AI visibility?

There's no evidence it directly improves rankings or AI citations today. Your visibility still depends on crawlability, content structure, authority, freshness, and off-site citations. llms.txt is a cheap optional add-on, not a substitute for those fundamentals.

How do I know if it's working?

Measure before and after. Watch whether AI crawlers actually request /llms.txt in your logs, and track whether your citation share in AI answers changes after you publish it. Because the providers haven't confirmed usage, real monitoring - not vendor promises - is the only honest way to judge the impact for your site.