--- title: "Blocking or Allowing 29 AI Crawlers, and the robots.txt Trap That Exposes You" url: https://hostmy.blog/block-ai-crawlers-robots-txt/ date: 2026-09-14 modified: 2026-09-03 lang: en author: "Aditya Sharma" description: "robots.txt matching is most-specific-wins, not additive. Give a crawler its own group and every wildcard rule you wrote silently stops applying to it." categories: - "RankReady" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1865-1024x538.jpg word_count: 1412 --- # Blocking or Allowing 29 AI Crawlers, and the robots.txt Trap That Exposes You Here is a robots.txt that does the opposite of what its author intended. `User-agent: * Disallow: /wp-admin/ Disallow: /checkout/ Disallow: /account/ Disallow: /?s= User-agent: GPTBot Disallow: /private/` The author believed GPTBot was now blocked from `/wp-admin/`, `/checkout/`, `/account/`, search results and `/private/`. GPTBot is blocked from `/private/` only. Everything else is open to it, including the checkout pages and every account URL. Nothing in that file looks wrong. No validator flags it. A site can publish that for months and nobody notices, because the exposure only shows up further along [the chain of requests between a question and your server](/ai-crawlers-request-by-request/). ## robots.txt matching is most-specific-wins, not additive This is the rule the whole post rests on, so it is worth stating plainly. A crawler reads robots.txt and selects exactly one group: the most specific `User-agent` line that matches its own name. It then obeys that group and ignores every other group in the file, including the wildcard one. Rules do not accumulate. The wildcard group is a fallback for crawlers with no named group, not a base layer that named groups extend. So the moment you write `User-agent: GPTBot` anywhere in the file, you have taken GPTBot out of your wildcard rules entirely. Whatever protection you thought applied to it stopped applying at that line. The same applies to `Bingbot`, `Googlebot` and every other named agent. Adding a named group is not a small addition. It is a complete replacement of that crawler's rule set. The way to catch it is [auditing your own file group by group](/audit-robots-txt-by-hand/) rather than trusting a validator. ## The fix is to repeat every rule inside every named group Correct version of the same file: `User-agent: * Disallow: /wp-admin/ Disallow: /checkout/ Disallow: /account/ Disallow: /?s= User-agent: GPTBot Disallow: /wp-admin/ Disallow: /checkout/ Disallow: /account/ Disallow: /?s= Disallow: /private/` Verbose, repetitive, and correct. Every named group must be self contained. That verbosity is the reason this goes wrong at scale. Managing 29 named groups by hand, each repeating your five baseline rules, means 145 lines that must all stay in sync. One rule added to the wildcard group and not copied down is a silent exposure. The most-specific-group trap in robots.txt The most-specific-group trap in robots.txt robots.txt as written User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Disallow: /?s= User-agent: GPTBot Allow: / The rules GPTBot actually applies User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Disallow: /?s= User-agent: GPTBot Allow: / GPTBot now has /wp-admin/, /cart/ and every search URL, which the file never intended. A named group replaces the wildcard group for that agent rather than adding to it, so writing a rule for GPTBot silently drops every rule you wrote for everyone. ## Three categories, and only one of them costs you answers Blocking is not one decision. AI crawlers do three different jobs, and treating them as one bucket is how sites accidentally remove themselves from the answers they were trying to appear in. | Category | What it does | What blocking it costs you | | -------- | ------------ | -------------------------- | | Answer and search crawlers | Build the index an assistant searches when a user asks a question | You disappear from that assistant's answers | | Training crawlers | Collect text for model training | Nothing immediate and nothing measurable | | User-triggered fetchers | Fetch a page because a person pasted the link | The user gets an error instead of your page | Most people who say "block AI crawlers" mean the training category. Blocking that category is a defensible position with a real argument behind it, and it costs you very little in visibility. [The argument for blocking, made properly](/should-you-block-ai-crawlers/), is worth reading before you write a rule you will not revisit. Blocking the answer category is a different decision with a real cost. Blocking the user-triggered category is almost never what anyone wants, since it breaks the case where someone deliberately shared your link. Read your existing rules against that table before changing anything. Sites that blanket-blocked everything in 2024 frequently blocked all three without noticing there were three. ## Names to have an opinion about The set worth managing runs to about 29 agents. Here are the ones where the decision actually matters. | User agent | Category | Common choice | | ---------- | -------- | ------------- | | `GPTBot` | Training | Depends on your position | | `OAI-SearchBot` | Answer | Allow | | `ChatGPT-User` | User-triggered | Allow | | `ClaudeBot` | Training | Depends on your position | | `Claude-SearchBot` | Answer | Allow | | `Claude-User` | User-triggered | Allow | | `PerplexityBot` | Answer | Allow | | `Perplexity-User` | User-triggered | Allow | | `Google-Extended` | Training | Depends on your position | | `CCBot` | Training | Depends on your position | | `Applebot-Extended` | Training | Depends on your position | | `Bytespider` | Training | Frequently blocked | Note what the first three rows show. One vendor operates separate agents for training, for search indexing and for direct user fetches. Writing `Disallow: /` under a single guess at that vendor's name gets you the wrong subset of the three. [The three OpenAI agents and which one you actually meant](/gptbot-wordpress/) is the clearest example of the confusion. ## robots.txt is a request, not a fence Worth being clear about what the file can and cannot do. robots.txt is a published preference. Well behaved crawlers honour it. It has no enforcement mechanism, and a crawler that ignores it faces no technical barrier. If you need an actual block, that lives at the server or the edge: a WAF rule, an nginx `if` on user agent, a Cloudflare bot rule. Those return 403 rather than asking politely, and since [Cloudflare's change to its crawler default](/cloudflare-crawler-default-wordpress/) some of them are on without anyone switching them on. The corollary catches people from the other direction. A crawler you allow in robots.txt can still be blocked by your host, your security plugin or your CDN, and robots.txt will not tell you. Check the actual response: `for ua in "GPTBot" "OAI-SearchBot" "ClaudeBot" "Claude-SearchBot" "PerplexityBot" "CCBot"; do code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://yoursite.com/) printf '%-18s %s\n' "$ua" "$code" done` Any `403` on a crawler you intended to allow is a block you did not write. That is layer one of [the manual audit this fits inside](/five-layer-ai-seo-audit/), and it invalidates everything above it. ## Find out who is actually fetching you Opinions about crawlers are cheap. Your access log is not. `awk -F'"' '{print $6}' /var/log/nginx/access.log \ | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Applebot|Bytespider|meta-externalagent|Amazonbot' \ | sort | uniq -c | sort -rn` On Apache the log path is usually `/var/log/apache2/access.log` and the field number may differ, so check the format first. Then look at what they got: `awk -F'"' '/GPTBot/ {split($3,a," "); print a[1]}' /var/log/nginx/access.log \ | sort | uniq -c | sort -rn` A column of `403` there tells you the rule is somewhere in your stack whether you wrote it or not. A column of `200` on `/wp-admin/` tells you the group problem at the top of this post is live on your site. ## Write the file in one shape and keep it that way Three rules make this manageable. Decide by category, not by vendor name. Pick your position on training, on answer indexing and on user fetches, then apply it consistently across all 29 agents. Knowing where that sits among the other [AI search jobs translated into WordPress terms](/ai-search-optimization-wordpress/) stops the file becoming the whole project. Never add a named group without copying your full baseline into it. If that feels like too much repetition to maintain by hand, that is the correct instinct, and it is the reason this is worth automating. Re-check after every plugin, host or CDN change. Security plugins ship crawler blocklists that turn on by default and quietly override whatever you decided. ## The five minute version on WordPress Generating self contained groups for 29 agents, keeping baseline rules synchronised across all of them, and separating training from answer from user-triggered is the kind of work that is tedious by hand and easy to get subtly wrong. That is what [RankReady](https://wordpress.org/plugins/rankready-ai-llm-seo/) manages. It is a WordPress AI SEO plugin, it runs alongside Rank Math, Yoast, AIOSEO or SEOPress rather than replacing them, and setup takes about five minutes. Most of that time is spent deciding your position on training crawlers, which is a policy question rather than a technical one. What it controls is crawlability, meaning who can fetch you and what they are asked not to touch. Whether an assistant then quotes you is a different question, and no plugin gets to answer it. ## Check your file for named groups today One command, and it tells you whether the trap applies to you: `curl -s https://yoursite.com/robots.txt | grep -i '^user-agent'` Every line that returns something other than `*` is a group that must repeat your baseline rules in full. Go and confirm each one does. Once your crawler rules are honest, the next question is what those crawlers receive when they arrive, which is covered in [Markdown endpoints and the cache trap under them](/markdown-version-of-blog-posts/). How many named user-agent groups does your robots.txt have, and does every one of them repeat your wildcard rules?