Does llms.txt
actually work?
We shipped the file on two sites, checked the logs, and read the two largest datasets on it. The file is cheap, harmless, and unread. The page that moved citations was the one the engines were already citing.
By Daria Dovzhikova · Updated September 2026
TL;DR
- Ahrefs, 137,210 domains, May 2026: 97% of llms.txt files got zero requests. 96% of the rest was bots.
- SE Ranking, nearly 300,000 domains: no relationship between the file and AI citations. Removing it made their model more accurate.
- Google: you do not need machine-readable files or AI text files to appear in AI features.
- Our two sites: no observed fetch of the file, and the one visibility gain we measured has a better explanation.
What the file is
llms.txt is a markdown file at the root of a site. Jeremy Howard proposed it on 3 September 2024: an H1 with the site name, a blockquote summary, and H2 sections of links with one-line notes, so an AI agent with a limited context window can read a curated summary instead of parsing navigation and a cookie banner. A companion llms-full.txt carries the whole site in one file. The spec got a v2 in August 2026. For coding agents that need documentation, it is a sensible idea. The question is whether the answer engines that decide who gets recommended ever open it.
Site one: this one
gtm-labs.co published /llms.txt on 25 May 2026 and /llms-full.txt on 1 June. 128 and 190 lines: the category definition, who the site is for, every service with a price, the FAQ.
The first finding was that we could not see whether anyone read it. Vercel, like most hosts, does not surface static-file requests in the dashboard, and web analytics only counts pages that run the tracking script, which a text file does not. The one runtime log query the platform allowed returned a single 24-hour window ending 16 September: 88 distinct paths received requests, and neither llms file was among them. Bots hit the homepage 146 times that day. Nothing asked for the file written for bots.
One day is not a study. It is exactly the visibility most teams have into this file, which is none. You publish it, you assume it works, and nothing in the stack can tell you otherwise.
Site two: Radar
Radar is an open-source Kubernetes UI by Skyhook, and a client. In August we put canonical positioning copy into llms.txt, llms-full.txt, and the site's JSON-LD in one pull request, merged 4 August. Two days earlier, a separate pull request had rewritten the /lens-alternative comparison page. An RSS feed shipped in the same fortnight.
We had a baseline. In July, across 48 unbranded prompts on four answer engines, the kind of question a platform engineer asks when nobody has handed them a product name, Radar was named in 3 answers. In September, 8 of 46.
That bundle cannot be decomposed. Three changes shipped inside a week and one re-measurement followed, so nothing in the data says which change moved the number. The data does say which change had a mechanism. Before the rewrite, the engines were already citing Radar's own comparison page, and citing it while recommending a competitor, because the page was a feature grid and the extraction pulled the competitor's row. The rewrite turned it into a sourced decision guide. That page had a visible causal path from "engine reads this" to "engine says that." The llms.txt file had no observed path at all: no log showed a fetch, no citation pointed at it, and no answer quoted a sentence that existed only there.
The full engagement is written up as a case study.
What the large datasets say
| Source | Sample | Finding |
|---|---|---|
| Ahrefs, server logs | 137,210 domains, May 2026. 28% had the file. | 97% of files got zero requests. Of the rest, 96% of requests were bots. Named AI crawlers were about a fifth of bot requests; OAI-SearchBot 0.74%. Verdict: "largely decoration." |
| SE Ranking, citation model | Nearly 300,000 domains. 10.13% had the file. | No meaningful relationship with AI citation frequency. Removing llms.txt as a variable made their XGBoost model more accurate on test data. |
| Google Search Central | Official documentation on AI features. | "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The AI optimization guide lists them under what you do not need to do. John Mueller compared llms.txt to the keywords meta tag. |
Two log-level studies, one modeling study, the engine with the most traffic, and our own two sites. Same answer.
Why it feels like it works
Writing a good llms.txt means writing a clear category definition, a one-line summary of what you sell, and a list of your best pages with a sentence each. That exercise improves the site whether or not anyone reads the file, and it usually leaks into the homepage and the schema. People credit the file for gains that came from finally writing the sentence.
Do this instead
Fix the pages the engines already cite
Run unbranded prompts, collect the citations, open the pages they point to. If one is yours and the engine still recommends a competitor, that page is feeding the wrong answer. This was Radar's biggest single leak.
Make the first 200 words extractable
A direct answer to the category question, the differentiators as short declarative sentences, then the comparison. Engines quote sentences. Give them ones worth quoting.
Put the canonical positioning in JSON-LD
Organization, Product or Service, FAQ. Structured data has a documented reader. llms.txt does not.
Confirm the retrieval crawlers can reach you
OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended each have a user agent, and a CDN bot rule can block them without anyone noticing. Check robots and WAF logs before you check visibility.
Earn mentions on the domains the engines cite
In every category measured this year, the top cited domains for a buyer question are not the vendors' own sites. LinkedIn, Reddit, a handful of niche blogs. Those are placements, and placements are work.
Keep the file if you have it. Free, harmless, and one day a coding agent may thank you. Just do not put it in the report as the reason the number moved. The free visibility check shows where you stand on unbranded prompts; the Growth Pilot does the five steps above as an experiment card, with a baseline in week one and the same read on day 28.
The minimal file, since you will make one anyway
# Your Company
> One sentence: what you sell, to whom, and the category name a buyer would use.
## Core pages
- [Pricing](https://example.com/pricing): Every tier with a number on it
- [Docs](https://example.com/docs): Start here for setup
- [Comparison](https://example.com/vs-competitor): Sourced, with the tradeoffs statedTen minutes. Then go fix the comparison page. Format per the proposal at llmstxt.org: H1, blockquote, H2 sections of links with optional notes after a colon.
FAQ
What is llms.txt?
A markdown file at the root of a website, proposed by Jeremy Howard in September 2024: an H1 with the site name, a blockquote summary, and H2 sections of links with one-line notes, so an AI agent with a limited context window can read a curated summary instead of parsing navigation and cookie banners. A companion llms-full.txt carries the complete content in one file.
Does llms.txt improve AI search visibility?
No evidence says so. Ahrefs pulled server logs from 137,210 domains for May 2026: 97% of llms.txt files received zero requests, and 96% of the requests that did occur came from bots. SE Ranking modeled nearly 300,000 domains and found no meaningful relationship between the file and AI citations; removing it as a variable made the model more accurate. Google's Search Central documentation says you do not need machine-readable files or AI text files to appear in AI features.
Does Google use llms.txt?
Google says it does not. The Search Central page on AI features states that you do not need to create new machine-readable files, AI text files, or markup to appear in those features, and the AI optimization guide lists such files under what you do not need to do. John Mueller compared llms.txt to the keywords meta tag and said Google was not pursuing it.
Does ChatGPT or Perplexity read llms.txt?
Rarely, on the evidence. In the Ahrefs data, named AI crawlers accounted for about a fifth of the requests to the 3% of files that got any traffic at all. OAI-SearchBot, the retrieval bot behind ChatGPT search, was 0.74% of those requests. In one 24-hour window of GTM Labs production logs, 88 distinct paths received requests and neither llms file was among them.
Should I remove my llms.txt file?
No. It is free, harmless, and useful to coding agents that need your documentation. Keep it. Stop reporting it as the reason a visibility number moved, and put the twenty minutes you would spend regenerating it into the comparison page the engines already cite.
What actually moves AI citations?
The pages the engines already cite, written so the extraction pulls your answer instead of a competitor's; the first 200 words of each commercial page structured as a direct answer; the same canonical positioning in JSON-LD; retrieval crawlers allowed through robots and the CDN; and third-party mentions on the domains the engines cite for your category, which in every category we have measured are not the vendors' own sites.
How do I generate an llms.txt file?
Write it by hand in ten minutes. An H1 with your company name, one blockquote sentence saying what you sell, to whom, and the category name a buyer would use, then an H2 section listing your pricing, docs, and comparison pages with one line each. A generator adds nothing a text editor does not, and the exercise of writing the summary sentence is the only part with a payoff.
Method notes. GTM Labs request data: Vercel runtime logs, production, static source, one 24-hour window ending 16 September 2026; the 7-day query timed out. Radar figures: GTM Labs AI-visibility baselines, July and September 2026, unbranded prompts across four engines, used with the client's portfolio-rights permission; no install or telemetry figures are used. Studies linked in the table above.
Related guides
Want the number before the opinion?
Week one of a pilot reads AI visibility on unbranded prompts across four engines. Thirty minutes to see whether a pilot fits.