---
title: "llms.txt Does Not Work, And We Generate It Anyway"
url: https://hostmy.blog/llms-txt-does-not-work/
date: 2026-09-05
modified: 2026-10-29
lang: en
author: "Aditya Sharma"
description: "Every study on llms.txt found no effect. Google compared it to the keywords meta tag. Here is why we still publish one on every site we run."
categories:
  - "Data Studies"
image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1838-1024x538.jpg
word_count: 1938
---

# llms.txt Does Not Work, And We Generate It Anyway

None of the studies we found shows llms.txt doing anything for the sites that publish it. SE Ranking looked at roughly 300,000 domains and, as [Search Engine Journal reported](https://www.searchenginejournal.com/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains/), found no significant correlation between citations and the file. Removing llms.txt from their model improved its accuracy, which is the statistical way of saying the variable was noise.

That is the finding. Anyone selling you an llms.txt generator as a citation lever is selling something this evidence does not support.

We generate the file anyway, on every WordPress site we run, as one small item on [the WordPress work AI search actually requires](/ai-search-optimization-wordpress/). The reasons are real, they are just not the reasons on the sales pages.

## Two studies and a forum comment, the same answer

The file is a Markdown index at `/llms.txt` that lists your important pages, so an AI model reading your site has a map instead of a crawl, and [producing one on WordPress](/llms-txt-generator-wordpress/) takes minutes. Lovely idea. The problem is that nothing on the receiving end agreed to read it.

| Source | What they looked at | What they found |
| ------ | ------------------- | --------------- |
| SE Ranking ([via SEJ, 20 Nov 2025](https://www.searchenginejournal.com/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains/)) | ~300,000 domains; file on 10.13% | No significant correlation with citations. Removing the variable improved model accuracy. |
| Ahrefs ([15 June 2026](https://ahrefs.com/blog/llmstxt-study/)) | 137,210 domains with traffic in May 2026; 28% publish the file | 97 percent of those files received zero traffic that month |
| John Mueller, Google ([via SEJ, April 2025](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag)) | A Reddit reply, not a Google policy | "AFAIK none of the AI services have said they're using LLMs.TXT"; "comparable to the keywords meta tag" |

Ahrefs is the number that should end the argument. Of the files it found, 97 percent received zero traffic in May 2026. That is not a weak signal. That is a signal you cannot distinguish from its absence. Ahrefs also cautions that its customers skew technical, so the 28 percent adoption figure is an upper bound, and that "fetched" doesn't mean "read".

Mueller's comparison is the one that stings, because the keywords meta tag is the standing joke of technical SEO. His explanation, in the same reply: "this is what a site-owner claims their site is about". Treat it as one Google engineer's forum comment, not as policy. Google's own [AI features page](https://developers.google.com/search/docs/appearance/ai-features) does not mention llms.txt, and it says you do not need new machine readable files or AI text files to appear in those features.

SE Ranking's authors were careful in the same way. They wrote that the file "doesn't seem to directly impact AI citation frequency. At least not yet." The "at least not yet" is doing real work there.

## Adoption is broad and shallow

Publishing the file is common among the sites Ahrefs measures: 28 percent of them have one. Being fetched is not. Because those sites skew technical, 28 percent is a ceiling for the web at large.

llms.txt: published against received any traffic
llms.txt: published against received any traffic
Domains publishing an llms.txt

28%
Whose file got any traffic

0.84%
Ahrefs, May 2026: of 137,210 domains, 28% publish the file; 97% of those files got zero traffic (3% of 28% = 0.84%).

Adoption is broad and shallow: far more sites publish the file than ever see a request for it.

There is a circularity that keeps this alive. Publishers say they will adopt once models read it. Models say they will read it once publishers adopt. Neither side moves, and tools keep shipping llms.txt as a headline feature because it is the easiest thing here to build.

## The three claims, priced

Read a few llms.txt product pages and the same three claims come back. Here is what each is worth against the evidence above.

**"Get cited by ChatGPT."** Mueller said in April 2025 that none of the AI services had said they use the file, and we found no later provider statement that changes that. In the Ahrefs sample 97 percent of the files got zero traffic. Worth zero until someone documents otherwise.

**"Help AI understand your site."** This one is slippery because it is unfalsifiable as written. Understand it how, measured against what? If the file is never fetched, nothing understood anything. Fetch logs settle it in an afternoon, and we have not seen the sellers of this show theirs.

**"Future-proof your site."** The only honest one of the three, and the reason we still publish. Optionality is a real thing to buy when it costs ten minutes. It stops being honest the moment it is priced like a growth feature.

A version of this file would be worth much more, and it is not a technical change. If a large publisher network adopted it and one model documented reading it, the circularity breaks. That has not happened.

## What we saw on our own sites

We run llms.txt on every site in the portfolio, and we read the access logs monthly. The fetch counts are low enough to count by hand. The clients that do request it are, overwhelmingly, the same coding agents that request Markdown, which suggests the file is riding along with the developer-tool wave rather than the consumer-AI one.

None of that is a citation claim. It is a crawl observation from our own logs, and anyone can check their own server in about five minutes with grep once they know [what a fetch looks like from the server side](/ai-crawlers-request-by-request/).

## So why do we ship it on every site

Four reasons, none of which are "it will get you cited".

**It costs almost nothing.** Generating the file is a few seconds of work once, then it maintains itself. Any cost-benefit calculation with a near-zero cost side tolerates a very small benefit side. The mistake is not publishing the file. The mistake is believing it did something.

**Writing one exposes your information architecture.** This is the part nobody talks about. An llms.txt forces you to name your twenty most important URLs and write one line describing each. Most site owners cannot do it. They open the page list, find 340 posts, and discover that four of them cover the same topic and none is obviously the canonical one. The exercise is worth more than the artifact.

**Optionality.** The file is a claim on a future that may not arrive. If adoption ever tips, sites that already publish are already done. If it never tips, the cost was ten minutes. That trade is fine as long as it is priced honestly, and pricing it honestly means you do not put llms.txt on your roadmap above things with measured effects.

**It travels with the part that does work.** This is the real reason, and it needs its own section.

## The Markdown half is the half with evidence behind it

llms.txt normally arrives bundled with a second capability: serving a clean Markdown version of each page to clients that ask for it, through HTTP content negotiation on the `Accept` header.

We have not found a published list of which AI clients send that header, so we make no claim about ChatGPT, Gemini or any other consumer product. What we can say comes from our own logs, above: the clients that request Markdown on our sites are coding agents. Anyone telling you Markdown negotiation gets you into ChatGPT answers is making the same category of claim as the llms.txt sellers.

Where a client does ask for it, the size difference is not subtle. One page we measured went from 4.7 MB of HTML to 18 KB of Markdown, roughly 260 times smaller for the same content. [Serving those copies without poisoning your cache](/markdown-version-of-blog-posts/) is the harder half.

| Format | Size | Ratio |
| ------ | ---- | ----- |
| Rendered HTML | 4.7 MB | 1x |
| Markdown equivalent | 18 KB | ~260x smaller |

Strip a page to Markdown and you have removed the nav, the footer, the cookie banner, three tracking scripts, the CSS, the related-posts widget and the comment form. What remains is the article. If a developer asks their agent about your documentation, that difference decides whether your page fits in the context window or gets truncated at the halfway mark.

Both of these are crawl-side facts. Neither is a promise that anything cites you.

## How to publish one without lying to yourself

If you are going to do it, do it properly and hold the expectations flat.

- **List 20 to 40 URLs, not 400.** The point is a curated map. A dump of your entire sitemap is a sitemap, and you already have one of those.

- **Write a real one-line description per URL.** "Blog post about hosting" helps nobody. Say what the page actually establishes.

- **Keep it current.** A stale index is worse than no index, because it advertises pages you have since merged or deleted.

- **Do not put it in your marketing.** No "AI-optimised" badge. No claim about visibility. The moment your homepage promises an outcome from this file, you have joined the people this article is about.

- **Measure the crawl, not the outcome.** Check your logs to see whether anything fetches `/llms.txt`. For most sites the honest answer will be almost nothing, and that is the correct data to have.

[RankReady](https://wordpress.org/plugins/rankready-ai-llm-seo/), our WordPress AI SEO plugin, generates the file and [the Markdown endpoints](/plugins/rankready/markdown-for-ai-wordpress/) together, which takes about five minutes to set up and runs alongside Rank Math, Yoast, AIOSEO or SEOPress without touching what they do. It is deliberately filed under crawl plumbing in the interface, not under a growth promise, because that is what the evidence supports. If you want the reasoning on what does move, the sentence-level data in [the length data that makes this concrete](/18-word-ceiling/) is a far better use of an afternoon.

## What would change my mind

A fair question to ask of anyone who takes a position like this one is what evidence would reverse it.

Three things would:

- One major AI provider documenting llms.txt as a real input, in writing, in their own docs rather than in a conference answer. [The free AI Readiness Checker](/plugins/rankready/ai-readiness-checker/) will show you whether your own file is served correctly in the meantime.

- A study finding a correlation that survives controls for content quality, domain age and existing rankings. The SE Ranking result is the opposite of that, since dropping the variable improved the model.

- Traffic to the files, not just their existence. Ahrefs found 97 percent of published files received none in May 2026.

None of those has happened. If one does, this article gets rewritten the same week, because the point of publishing numbers is that you are bound by them when they change.

Related: [llms.txt on a multilingual WordPress site](/multilingual-wordpress-llms-txt/) and [five curl checks for your llms.txt](/llms-txt-validator-wordpress/).

## The pattern underneath this

llms.txt is one instance of something that keeps happening in AI search tooling. A file format gets proposed, the tooling ships before the demand exists, and a feature gets sold on an outcome nobody has measured, which is most of [the dishonest half of the AI SEO pitch](/what-ai-seo-means/).

The same shape shows up in the audit tools. Fourteen free agent-readiness checkers now exist and [not one of them reads your content](/ai-readiness-checkers/), which does not stop any of them from giving you a score. It shows up in crawler access too, where the file that says you allow GPTBot and [the server that returns 429 to GPTBot](/host-blocking-ai-crawlers/) can disagree completely without anyone noticing.

The common failure is trusting the declaration instead of the behaviour. llms.txt is a declaration. A 200 response with clean Markdown, logged, from a crawler that actually asked for it, is behaviour.

Publish the file. Spend the ten minutes. Then go and look at what your server is really doing when a crawler shows up, because that is where the findings are.

What would you need to see, specifically, before you would put llms.txt above sentence-level editing on your own list?