--- title: "llms.txt Does Not Work, And We Generate It Anyway" url: https://hostmy.blog/llms-txt-does-not-work/ date: 2026-09-05 modified: 2026-09-03 lang: en author: "Aditya Sharma" description: "Every study on llms.txt found no effect. Google compared it to the keywords meta tag. Here is why we still publish one on every site we run." categories: - "Data Studies" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1838-1024x538.jpg word_count: 1937 --- # llms.txt Does Not Work, And We Generate It Anyway Not one published study has found llms.txt doing anything for the sites that publish it. SE Ranking looked at roughly 300,000 domains and found no correlation with AI visibility. Removing llms.txt from their model actually made the model more accurate, which is the statistical way of saying the variable was noise. That is the finding. Every vendor selling you an llms.txt generator as a citation lever is selling something the evidence does not support, and most of them know it. We generate the file anyway, on every WordPress site we run, as one small item on [the WordPress work AI search actually requires](/ai-search-optimization-wordpress/). The reasons are real, they are just not the reasons on the sales pages. ## Four separate looks, the same answer The file is a Markdown index at `/llms.txt` that lists your important pages, so an AI model reading your site has a map instead of a crawl, and [producing one on WordPress](/llms-txt-generator-wordpress/) takes minutes. Lovely idea. The problem is that nothing on the receiving end agreed to read it. | Source | What they looked at | What they found | | ------ | ------------------- | --------------- | | SE Ranking | ~300,000 domains | No correlation with AI visibility. Dropping the variable improved model accuracy. | | Ahrefs | 137,000 domains, about 38,000 with llms.txt | 97 percent of those files never requested once | | Google | Public statements | Says it does not use llms.txt, and compared it to the keywords meta tag | | Perplexity, OpenAI, Bing, Copilot | Published documentation | No stated support from any of them | Ahrefs is the number that should end the argument. Of the roughly 38,000 sites carrying the file, 97 percent were never asked for it once. That is not a weak signal. That is a signal you cannot distinguish from its absence. Google's comparison is the one that stings, because the keywords meta tag is the standing joke of technical SEO. It was self-reported, it was gamed within months, and search engines stopped reading it decades ago. Google put llms.txt in that category on purpose. Cyrus Shepard, who worked as a Google quality rater, put it plainly: "You don't even need an llms.txt file. At least not yet." The "at least not yet" is doing real work there. ## Adoption is a rounding error, and the top of the web is not participating Only a minority of sites have published an llms.txt. Among the largest sites it is rarer still. That second number is the one to think about. Standards do not spread from the bottom. HTTPS, structured data, sitemaps, mobile viewports: every one of them was pushed down by large sites and platform defaults, not up from small ones. When the largest sites on the web have overwhelmingly declined to publish a file that takes ten minutes to write, they are telling you something about their read on the return. llms.txt: published against ever fetched llms.txt: published against ever fetched Domains publishing an llms.txt 28% Whose file was ever fetched 0.84% Ahrefs, May 2026: of 137,000 domains, 28% publish the file and 97% of those were never requested. Adoption is broad and shallow. The sites with the most to gain from machine access are the ones least likely to have published the file. There is a circularity that keeps this alive. Publishers say they will adopt once models read it. Models say they will read it once publishers adopt. Neither side moves, and tools keep shipping llms.txt as a headline feature because it is the easiest thing here to build. ## The three claims, priced Read a few llms.txt product pages and the same three claims come back. Here is what each is worth against the evidence above. **"Get cited by ChatGPT."** OpenAI publishes no support for the file. The claim has no documented mechanism, and in the Ahrefs sample 97 percent of the files published were never fetched. Worth zero until someone documents otherwise. **"Help AI understand your site."** This one is slippery because it is unfalsifiable as written. Understand it how, measured against what? If the file is never fetched, nothing understood anything. Fetch logs settle it in an afternoon, and almost nobody selling this shows you theirs. **"Future-proof your site."** The only honest one of the three, and the reason we still publish. Optionality is a real thing to buy when it costs ten minutes. It stops being honest the moment it is priced like a growth feature. A version of this file would be worth much more, and it is not a technical change. If a large publisher network adopted it and one model documented reading it, the circularity breaks. That has not happened. ## What we saw on our own sites We run llms.txt on every site in the portfolio, and we read the access logs monthly. The fetch counts are low enough to count by hand. The clients that do request it are, overwhelmingly, the same coding agents that request Markdown, which suggests the file is riding along with the developer-tool wave rather than the consumer-AI one. That matches the split in the negotiation data. Coding agents fetch machine-friendly formats because reading source material efficiently is their job. Consumer AI browsing fetches the page a browser would fetch. None of that is a citation claim. It is a crawl observation from our own logs, and anyone can check their own server in about five minutes with grep once they know [what a fetch looks like from the server side](/ai-crawlers-request-by-request/). ## So why do we ship it on every site Four reasons, none of which are "it will get you cited". **It costs almost nothing.** Generating the file is a few seconds of work once, then it maintains itself. Any cost-benefit calculation with a near-zero cost side tolerates a very small benefit side. The mistake is not publishing the file. The mistake is believing it did something. **Writing one exposes your information architecture.** This is the part nobody talks about. An llms.txt forces you to name your twenty most important URLs and write one line describing each. Most site owners cannot do it. They open the page list, find 340 posts, and discover that four of them cover the same topic and none is obviously the canonical one. The exercise is worth more than the artifact. **Optionality.** The file is a claim on a future that may not arrive. If adoption ever tips, sites that already publish are already done. If it never tips, the cost was ten minutes. That trade is fine as long as it is priced honestly, and pricing it honestly means you do not put llms.txt on your roadmap above things with measured effects. **It travels with the part that does work.** This is the real reason, and it needs its own section. ## The Markdown half is the half with evidence behind it llms.txt normally arrives bundled with a second capability: serving a clean Markdown version of each page to clients that ask for it, through HTTP content negotiation on the `Accept` header. That one has a verified user list. Coding agents honour it: - Claude Code - GitHub Copilot Chat and Copilot CLI - Cursor - Microsoft Copilot - OpenClaw - OpenCode And a list that does not: - ChatGPT browsing - Claude.ai - Perplexity - Gemini - Grok So the split is real and it is narrow. Coding agents fetch Markdown. Consumer AI browsing does not. Anyone telling you Markdown negotiation gets you into ChatGPT answers is making the same category of claim as the llms.txt sellers. Where it does apply, the size difference is not subtle. One page we measured went from 4.7 MB of HTML to 18 KB of Markdown, roughly 260 times smaller for the same content. [Serving those copies without poisoning your cache](/markdown-version-of-blog-posts/) is the harder half. | Format | Size | Ratio | | ------ | ---- | ----- | | Rendered HTML | 4.7 MB | 1x | | Markdown equivalent | 18 KB | ~260x smaller | Strip a page to Markdown and you have removed the nav, the footer, the cookie banner, three tracking scripts, the CSS, the related-posts widget and the comment form. What remains is the article. If a developer asks their agent about your documentation, that difference decides whether your page fits in the context window or gets truncated at the halfway mark. Both of these are crawl-side facts. Neither is a promise that anything cites you. ## How to publish one without lying to yourself If you are going to do it, do it properly and hold the expectations flat. - **List 20 to 40 URLs, not 400.** The point is a curated map. A dump of your entire sitemap is a sitemap, and you already have one of those. - **Write a real one-line description per URL.** "Blog post about hosting" helps nobody. Say what the page actually establishes. - **Keep it current.** A stale index is worse than no index, because it advertises pages you have since merged or deleted. Stale content is a measurable problem in its own right, and the citation data backs that up: pages that go stale are around three times more likely to lose citations they previously had. - **Do not put it in your marketing.** No "AI-optimised" badge. No claim about visibility. The moment your homepage promises an outcome from this file, you have joined the people this article is about. - **Measure the crawl, not the outcome.** Check your logs to see whether anything fetches `/llms.txt`. For most sites the honest answer will be almost nothing, and that is the correct data to have. [RankReady](https://wordpress.org/plugins/rankready-ai-llm-seo/), our WordPress AI SEO plugin, generates the file and the Markdown endpoints together, which takes about five minutes to set up and runs alongside Rank Math, Yoast, AIOSEO or SEOPress without touching what they do. It is deliberately filed under crawl plumbing in the interface, not under a growth promise, because that is what the evidence supports. If you want the reasoning on what does move, the sentence-level data in [the length data that makes this concrete](/18-word-ceiling/) is a far better use of an afternoon. ## What would change my mind A fair question to ask of anyone who takes a position like this one is what evidence would reverse it. Three things would: - One major AI provider documenting llms.txt as a real input, in writing, in their own docs rather than in a conference answer. - A study finding a correlation that survives controls for content quality, domain age and existing rankings. The SE Ranking result is the opposite of that, since dropping the variable improved the model. - Adoption crossing meaningfully into the top tier of sites. From that base, that is a long walk. None of those has happened. If one does, this article gets rewritten the same week, because the point of publishing numbers is that you are bound by them when they change. ## The pattern underneath this llms.txt is one instance of something that keeps happening in AI search tooling. A file format gets proposed, the tooling ships before the demand exists, and a feature gets sold on an outcome nobody has measured, which is most of [the dishonest half of the AI SEO pitch](/what-ai-seo-means/). The same shape shows up in the audit tools. Fourteen free agent-readiness checkers now exist and [not one of them reads your content](/ai-readiness-checkers/), which does not stop any of them from giving you a score. It shows up in crawler access too, where the file that says you allow GPTBot and [the server that returns 429 to GPTBot](/host-blocking-ai-crawlers/) can disagree completely without anyone noticing. The common failure is trusting the declaration instead of the behaviour. llms.txt is a declaration. A 200 response with clean Markdown, logged, from a crawler that actually asked for it, is behaviour. Publish the file. Spend the ten minutes. Then go and look at what your server is really doing when a crawler shows up, because that is where the findings are. What would you need to see, specifically, before you would put llms.txt above sentence-level editing on your own list?