---
title: "Audit Your Own robots.txt for the Most-Specific-Match Trap, by Hand"
url: https://hostmy.blog/audit-robots-txt-by-hand/
date: 2026-09-21
modified: 2026-09-21
lang: en
author: "Aditya Sharma"
description: "robots.txt matching is most-specific-wins, not additive. Add one named crawler group and your global rules silently stop applying to it."
categories:
  - "WordPress"
image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1886-1024x538.jpg
word_count: 1661
---

# Audit Your Own robots.txt for the Most-Specific-Match Trap, by Hand

Almost everyone reads robots.txt wrong, and the way they read it wrong is consistent.

They read it top to bottom and assume the rules accumulate. Wildcard rules apply to everyone, then named crawlers get extra rules on top. That is a reasonable guess and it is not how the file works.

robots.txt matching is most specific wins. A crawler finds the single group whose user agent line best matches its own name, obeys that group, and ignores every other group in the file. Including the wildcard group. Including all the rules you carefully wrote there.

So the moment you add a two line block for one named crawler, every global `Disallow` stops applying to that crawler. Silently. No warning, no validator complaint, no change in Search Console. The only place it surfaces is in [the ordinary HTTP requests behind an AI citation](/ai-crawlers-request-by-request/), one status code at a time.

How a crawler picks its robots.txt group

How a crawler picks its robots.txt group

Read every group
each User-agent line

Pick the most
specific match
an exact token beats *

Apply that group
only
the others are discarded

parse

one winner
Adding a group for a named agent removes the wildcard rules from that
agent, it does not extend them.

The algorithm selects exactly one group and ignores the rest. That single step is where most hand-written AI crawler rules quietly go wrong.

## The shape of the bug

Here is a robots.txt that looks completely reasonable:

`User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=

User-agent: GPTBot
Crawl-delay: 10

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap_index.xml`

The author's intent is clear. Everyone stays out of admin, cart, checkout and search results. GPTBot additionally slows down. CCBot is blocked entirely.

What actually happens is that GPTBot matches its own group, obeys `Crawl-delay: 10`, and is now free to crawl `/wp-admin/`, `/cart/`, `/checkout/` and every search results URL on the site. Those four rules apply to nobody except crawlers with no named group of their own. Worth knowing too that [GPTBot has three sibling agents](/gptbot-wordpress/), and a group naming only one of them says nothing about the other two.

CCBot has the same problem in a form that does not matter, because its own group blocks everything anyway.

## Reading it by hand, in four passes

You can catch this without any tooling. Four passes over the file.

**Pass one: list the groups.** A group starts at a `User-agent:` line and runs until the next `User-agent:` line that follows at least one rule. Consecutive `User-agent:` lines with no rule between them stack into one group sharing the rules underneath.

`curl -sSL https://yoursite.com/robots.txt | grep -n -i '^user-agent:'`

Expected output:

`1:User-agent: *
7:User-agent: GPTBot
10:User-agent: CCBot`

Three groups. Note the `-L`, without it a redirect gives you an empty file and a false clean bill of health.

**Pass two: for each named group, ask what it lost.** Take the rules under `User-agent: *` and read them again as the list of protections that named crawler no longer has. That reframing is the whole trick. The wildcard group is not a baseline. It is the fallback for crawlers you did not name.

**Pass three: check whether the named group is doing anything.** A group containing only `Crawl-delay` is the classic case. It carries almost no instruction and costs the crawler every global rule you wrote.

**Pass four: confirm `Sitemap:` sits outside every group.** Sitemap directives are global regardless of position, but putting one inside a group misleads the next human who reads the file.

## The script that does the same four passes

Save this as `whichgroup.py`. Standard library only.

`#!/usr/bin/env python3
"""Show which robots.txt group a named crawler actually obeys."""
import sys

def parse(lines):
"""Return [(agents, rules)] in file order. A rule then a new agent ends a group."""
groups, agents, rules = [], [], []
for raw in lines:
line = raw.split("#", 1)[0].strip()
if not line or ":" not in line:
continue
field, value = line.split(":", 1)
field, value = field.strip().lower(), value.strip()
if field == "user-agent":
if rules:
groups.append((agents, rules))
agents, rules = [], []
agents.append(value)
elif field in ("allow", "disallow", "crawl-delay"):
rules.append((field, value))
if agents or rules:
groups.append((agents, rules))
return groups

def pick(groups, ua):
"""Most specific wins, and RFC 9309 requires every matching group to merge."""
ua_l = ua.lower()
best_len = -1
for agents, _ in groups:
for a in agents:
a_l = a.lower()
if a_l != "*" and a_l in ua_l:
best_len = max(best_len, len(a_l))
named, named_rules, star, star_rules = [], [], [], []
for agents, rules in groups:
toks = [a.lower() for a in agents]
if best_len > 0 and any(t != "*" and t in ua_l and len(t) == best_len for t in toks):
named += [a for a in agents if a not in named]
named_rules += rules
if "*" in toks:
star += [a for a in agents if a not in star]
star_rules += rules
return (named, named_rules) if named else None, (star, star_rules) if star else None

def main(path, ua):
lines = sys.stdin.readlines() if path == "-" else open(path, encoding="utf-8").readlines()
groups = parse(lines)
if not groups:
print("no groups parsed, robots.txt is empty or malformed")
return 1
best, star = pick(groups, ua)
chosen = best or star
print("crawler %s" % ua)
print("groups in file %d" % len(groups))
if best:
print("matched group User-agent: %s" % ", ".join(best[0]))
if star:
print("IGNORED group User-agent: * (%d rule(s) do NOT apply to %s)"
% (len(star[1]), ua))
elif star:
print("matched group User-agent: * (no named group for %s)" % ua)
else:
print("matched group none, %s is unrestricted" % ua)
return 0
print("\nrules %s actually obeys:" % ua)
for field, value in chosen[1]:
print(" %-11s %s" % (field.capitalize() + ":", value or "(empty)"))
if best and star:
print("\nrules it silently drops:")
for field, value in star[1]:
print(" %-11s %s" % (field.capitalize() + ":", value or "(empty)"))
return 0

if __name__ == "__main__":
if len(sys.argv) != 3:
print("usage: python3 whichgroup.py robots.txt GPTBot")
print(" curl -sSL https://site.com/robots.txt | python3 whichgroup.py - GPTBot")
sys.exit(2)
sys.exit(main(sys.argv[1], sys.argv[2]))`

Run it against the example file above:

`python3 whichgroup.py robots.txt GPTBot`

```
crawler GPTBot
groups in file 3
matched group User-agent: GPTBot
IGNORED group User-agent: * (4 rule(s) do NOT apply to GPTBot)

rules GPTBot actually obeys:
Crawl-delay: 10

rules it silently drops:
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=
```

And against a crawler with no group of its own:

`crawler ClaudeBot
groups in file 3
matched group User-agent: * (no named group for ClaudeBot)

rules ClaudeBot actually obeys:
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=`

Two crawlers, one file, completely different rule sets. That is the mechanism, made visible. Group resolution is one layer of [a five layer AI SEO audit run with curl](/five-layer-ai-seo-audit/), and a correct file sitting above a 403 still gets you nothing.

## It happens on real sites

Point it at a live file:

`curl -sSL https://yoursite.com/robots.txt | python3 whichgroup.py - GPTBot`

Run against wordpress.org in September 2026, the output was:

`crawler GPTBot
groups in file 12
matched group User-agent: GPTBot
IGNORED group User-agent: * (7 rule(s) do NOT apply to GPTBot)

rules GPTBot actually obeys:
Allow: /

rules it silently drops:
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-admin/load-scripts.php
Allow: /wp-admin/load-styles.php
Disallow: /search
Disallow: /?s=
Disallow: /plugins/search/`

In that case the outcome is harmless, since the intent was clearly to allow GPTBot everywhere. Whether that intent is the right one is a separate argument, put both ways in [the case each way on letting AI crawlers in](/should-you-block-ai-crawlers/). But the structure is identical to the version that hurts. Change the wildcard group to protect something real and the named group would walk straight past it.

## Fixing it

There are only two safe patterns.

**Repeat the rules.** If a crawler needs a named group, copy every wildcard rule into that group as well. Verbose, ugly, correct.

`User-agent: GPTBot
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=`

**Or do not create the group.** If the only thing you want to say to a crawler is already in the wildcard group, leave it there. A named group with one throwaway directive is the worst of both worlds.

Then re-run the script for every crawler you named, plus one you did not, to confirm the fallback still behaves. The full agent list, and what each category actually costs you, is in [blocking or allowing the 29 named AI crawlers](/block-ai-crawlers-robots-txt/).

## The rules that are not group scoped

Two things behave differently and are worth knowing.

`Sitemap:` is global. It applies wherever it sits in the file.

And blocking a crawler in robots.txt does not stop it reaching a page linked from elsewhere. robots.txt is a crawl instruction, not an access control. If a URL must not be fetched, refuse it at the server. Whether the request even arrives is a separate question, answered in [five curl commands that prove reachability](/ai-crawler-test/), and whether it arrived last week is answered in [reading an access log for AI crawler traffic](/access-log-ai-crawler-analytics/).

## If you would rather not run scripts

Hand auditing is worth doing once so the rule stops being surprising. Maintaining it across 29 crawler agents by hand is a different matter, and it is exactly where this bug gets reintroduced.

RankReady is a WordPress AI SEO plugin that writes robots.txt groups with the rules repeated per named crawler, so a named group never drops your global rules. It also shows you the resolved view: what each crawler ends up obeying, rather than what the file appears to say.

It runs alongside Rank Math, Yoast, AIOSEO and SEOPress. Those own titles, meta and sitemaps; this owns crawler access and machine readability, so there is nothing to migrate. Setup takes about five minutes.

The claim is narrow on purpose: correct robots.txt groups mean crawlers can reach what you meant them to reach. That is crawlability, not a promise about citations. It is on [WordPress.org](https://wordpress.org/plugins/rankready-ai-llm-seo/).

Open your own robots.txt and count the named groups. For each one, can you say out loud which global rules it just threw away?