What is llms.txt, in plain terms?
Think of llms.txt as the language-model cousin of two files you already know: robots.txt and sitemap.xml. robots.txt tells crawlers what they may not access. sitemap.xml lists every URL you have. llms.txt is meant to say something different: "here are the pages that actually matter, in clean markdown, with a one-line description of each — read these first."
The idea was proposed by Jeremy Howard (co-founder of Answer.AI and fast.ai) in September 2024 and documented at llmstxt.org. The motivating problem is genuine: language models work inside limited context windows, and a modern web page is mostly navigation, scripts, cookie banners, and boilerplate wrapped around a little bit of substance. The proposal's argument is that a curated, markdown index lets a model grab your canonical answers — your service definitions, your pricing page, your FAQs — without wading through the clutter or guessing.
So the concept is reasonable. The catch, which most explainer pages bury, is that a proposal only matters if the engines choose to honor it — and so far, they largely have not.
What does an llms.txt file look like? (the proposed spec)
The format is deliberately minimal. A valid file, per the spec, is just markdown with:
An optional companion file, llms-full.txt, contains the full flattened text of your key pages in a single document, so a model can ingest everything in one request. Here is a stripped-down llms.txt example for a service business:
# AISEO USA
> AI SEO agency for US businesses — Google rankings plus
> citations in ChatGPT, Perplexity, Gemini, and AI Overviews.
## Services
- [AI SEO Services](https://aiseousa.com/ai-seo-services): Full-program AI search optimization
- [GEO Services](https://aiseousa.com/generative-engine-optimization-services): Get cited by generative engines
## Resources
- [Free AI Visibility Audit](https://aiseousa.com/free-ai-visibility-audit): What AI says about your business
There is no registration, no gatekeeper, and no validation authority. You publish it at your root the same way you publish robots.txt, and that is the entire installation. It takes about an hour if your content is already organized. That low cost is the strongest thing anyone can honestly say for it.
How is llms.txt different from robots.txt and sitemap.xml?
The three files are easy to confuse because they all live at your root and all speak to bots. They do very different jobs — and only one of them is a real, enforced standard.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Block or allow crawlers | Enumerate all URLs | Curate key content for LLMs |
| Format | Directives | XML | Markdown |
| Audience | All bots | Search engines | AI assistants and agents |
| Official standard? | Yes (Robots Exclusion Protocol) | Yes (sitemaps.org) | No — a proposal, not adopted by any major engine |
| Confirmed usage | Universally respected | Universally used | Not confirmed by OpenAI, Google, Anthropic, or Perplexity |
That last row is the part most agencies skip when they sell "llms.txt optimization." Unlike robots.txt — which every serious crawler obeys — llms.txt is a suggestion no major AI provider has agreed to follow. Understanding llms.txt vs robots.txt is really understanding that one is a rule and the other is a request no one has said yes to yet.
Does llms.txt actually work? The honest data
This is the real question behind "what is llms.txt," so here is the evidence without the spin.
Ahrefs studied 137,210 domains and found that 97% of llms.txt files received zero requests from AI crawlers in a single month (Ahrefs llms.txt study, May 2026). Not "a few." Zero. The report's blunt summary is that no major AI platform has ever committed to reading the file — and the logs bear that out, with everyday tools like Slackbot fetching the file more often than PerplexityBot did.
Google has said the quiet part out loud. Google Search Advocate John Mueller compared llms.txt to the old, abandoned keywords meta tag: "AFAIK none of the AI services have said they're using llms.txt (and you can tell when you look at your server logs that they don't even check for it) ... this is what a site-owner claims their site is about. ... At that point, why not just check the site directly?" (via Search Engine Journal). His deeper point is a trust problem: a self-declared index is trivially easy to game — show one set of pages to the AI and another to users — so a careful engine has to re-check the real content anyway, which defeats the purpose.
"llms.txt is a lottery ticket that costs an hour. Buy the ticket if you like — but don't confuse it with the actual work of getting cited. In the audits we run, not one site earned an AI citation because of its llms.txt. They earned citations because their pages were clear, sourced, and reachable." — Thomas, Founder of AISEO USA, 16 years in digital marketing
So, does llms.txt work? As a retrieval or citation shortcut, the honest answer today is no. It is an unproven, largely un-consumed file. That is not a prediction that it will always be useless — proposals sometimes get adopted — but you should treat it as a cheap bet on the future, never as a strategy for the present.
Do ChatGPT and Google actually use llms.txt?
No major AI company has confirmed it. OpenAI (ChatGPT), Google (Gemini and AI Overviews), Anthropic (Claude), and Perplexity have all declined to announce support for the standard. When Ahrefs looked at who was actually requesting these files, AI retrieval bots accounted for a tiny fraction of the traffic — the file simply is not part of how the mainstream engines find and cite content.
There is a narrow exception worth naming honestly: some developer-documentation tools and coding agents do read llms.txt and llms-full.txt, because ingesting a clean, flattened doc set saves tokens. If you run a large documentation site or a dev-tools product, the 3% of files that do get requests skew toward you. For a typical local or B2B service business, the practical answer is that the engines your customers use are not reading it.
The takeaway is not "never publish one." It is "know what you are getting." An llms.txt will not put you in an AI answer. What puts you there is a different, well-documented set of levers.
So why publish one at all?
Given all of that, we still ship a clean llms.txt inside our engagements — for three defensible reasons, none of which involve promising it will get you cited:
- It is nearly free. A good one takes about an hour if your pages are already organized. Near-zero cost, non-zero option value, no downside risk.
- A few readers do exist today. Dev tools, coding agents, and some retrieval pipelines read it now, and agentic browsing is the direction of travel. The file is a low-stakes bet on where machine readers are heading.
- It forces editorial clarity. Writing one makes you decide which 15–30 pages actually define your business and state, in one line, what each answers. That exercise improves your site everywhere — and a well-curated file, not a renamed sitemap, is the version with any chance of being useful.
If a vendor quotes you four figures for "an llms.txt deployment," walk. It is a text file, not a growth channel.
What to do instead of relying on llms.txt
Put your budget where the causal evidence actually points. In the AI-visibility audits we run, the sites that get cited share a set of traits — and none of them is the presence of an llms.txt file. In rough priority order:
robots.txt. This file is universally respected, and we routinely find AI bots quietly blocked on sites that then wonder why they are invisible.That list is the substance of real generative engine optimization, and it is the core of any serious LLM SEO or full AI SEO services program. llms.txt is one optional line item inside it, priced accordingly — an hour of work, published without illusions, while the real effort goes into access, clarity, evidence, and corroboration. And it comes with the same honesty we give every client: no one can guarantee AI citations. Anyone who does is lying. What good work does is raise the probability and show you the receipts every month.
Frequently Asked Questions
What is llms.txt in simple terms?
llms.txt is a plain markdown file placed at your site's root (yoursite.com/llms.txt) that lists your most important pages with short descriptions, so AI language models can find your best content without wading through navigation and code. Proposed by Jeremy Howard in 2024, it is best understood as a curated table of contents written for LLMs — a voluntary suggestion, not an enforced standard.
Does llms.txt actually work?
As a way to get cited by AI, the honest answer today is no. Ahrefs studied 137,210 domains and found 97% of these files received zero requests from AI crawlers in a month (Ahrefs), and no major AI provider has confirmed using it. It is a cheap, low-risk bet on the future — not a working citation channel in 2026.
Do ChatGPT and Google use llms.txt?
Not that either has confirmed. OpenAI, Google, Anthropic, and Perplexity have all declined to announce support for the standard. Google's John Mueller compared it to the discredited keywords meta tag and noted the AI services don't even check for the file in server logs (Search Engine Journal). Some developer-doc and coding tools do read it, but the mainstream consumer engines do not.
Is llms.txt the same as robots.txt?
No — they are close to opposites. robots.txt tells crawlers what they may not access and is a real, universally respected standard. llms.txt suggests what an AI should read first and is an unadopted proposal. Understanding llms.txt vs robots.txt comes down to this: one is a rule engines obey, the other is a request no major engine has agreed to.
How do I create an llms.txt file?
Write a markdown file with an H1 for your brand name, a blockquote summarizing what you do, then H2 sections listing 15–30 key pages, each with a one-sentence description. Upload it to your site's root directory. Optionally add llms-full.txt with the full text of those pages. Curate ruthlessly — a file that just mirrors your sitemap tells a model nothing new.
Will llms.txt get me cited in ChatGPT?
Not by itself, and not because of the file. Citations come from crawler access, answer-first content, sourced evidence, entity clarity, and third-party corroboration — not from an index a majority of engines never request. Publish it if you like the cheap upside, but expect nothing from it on its own. It is one small, optional piece of generative engine optimization.
Should my business publish an llms.txt file?
Yes, with correct expectations — it costs about an hour and carries no downside. Publish it especially if you run a documentation-heavy or developer-tools site, where the few AI agents that read the file actually visit. Just don't let it substitute for the work that moves AI visibility, and don't pay a premium for "deployment" of a plain text file.
What matters more than llms.txt for AI visibility?
Everything with confirmed causal weight: open AI-crawler access in robots.txt, answer-first pages, evidence density (statistics and quotes lifted visibility up to 40% in the founding GEO study — arXiv), consistent entity and schema, and third-party mentions. With top-10 rankings and AI citations now overlapping only 38% of the time (Ahrefs), those levers — not a text file — are what get you named.