---
title: "Your Host May Be Blocking AI Crawlers Without Telling You"
url: https://hostmy.blog/host-blocking-ai-crawlers/
date: 2026-09-07
modified: 2026-09-03
lang: en
author: "Aditya Sharma"
description: "We found our own server returning 429 to GPTBot while robots.txt said Allow. Here is how to test yours."
categories:
  - "Data Studies"
image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1842-1024x538.jpg
word_count: 1916
---

# Your Host May Be Blocking AI Crawlers Without Telling You

Testing our own site, the server was returning 429 to GPTBot on every request while its robots.txt said `Allow: /`. The file granted permission. The server refused the connection. Nobody involved knew, because nothing inside WordPress can see it.

We reproduced it three times out of three. Same user agent, same URLs, same result.

The cause was a firewall rule at the hosting layer, on a LiteSpeed server, applied above WordPress. No plugin caused it. No plugin could have detected it, and no plugin can override it.

## What the responses actually looked like

The pattern is the interesting part, because it is specific enough to fingerprint.

| Path requested as GPTBot | Status |
| ------------------------ | ------ |
| Homepage | 429 |
| Post URLs | 429 |
| Static assets, CSS and images | 429 |
| /robots.txt | 200 |
| /sitemaps.xml | 200 |

Everything blocked, except the two files that describe what is allowed.

That exemption is not an accident. Whoever wrote the rule deliberately let robots.txt and the sitemap through. The likely intent was to look cooperative while shedding the traffic. The effect is that every automated check of this site passes.

Consider what that does to the tools people use. A checker fetches robots.txt, sees `Allow`, reports green. It never requests a real page as GPTBot, so it never sees the 429. The site owner gets a clean bill of health for a site that is fully closed to the crawler.

Static assets returning 429 is the detail that confirms it is a blanket rule at the edge rather than anything content-aware. Whatever made that decision sat in front of everything.

Where the refusal happens when the host does it

Where the refusal happens when the host does it

AI crawler
GET /post/

Host edge filter
bot rule

WordPress
never reached

Accept: text/html

not forwarded

429 or 403
returned here
origin never sees it
Your robots.txt and sitemap.xml still return 200, which is why the site looks fine.

robots.txt is served by WordPress and the refusal is issued before WordPress runs, so the file says Allow and the crawler still leaves empty handed.

## Why 429 is worse than 403

A 403 says no. A 429 says too many requests, slow down and come back.

For a crawler, those are different instructions. A 429 is a rate limit, so a well-behaved bot backs off and retries later. It does not conclude it is banned, and it does not report the site as blocked anywhere the owner would see. It just quietly fetches your site less and less often.

That makes 429 the failure mode least likely to be noticed. There is no error page, no console warning, no email. The only place it exists is in server logs most site owners have never opened, which is why [reading a server log for AI crawler hits by hand](/read-server-log-ai-crawlers/) is the one habit that catches it.

## Why nothing inside WordPress can catch this

WordPress sees requests that reach WordPress. That is the whole problem.

A firewall rule at the host, a rule in an edge network, or a rule in the web server configuration all execute before PHP starts. If the rule returns 429, the request never becomes a WordPress request. There is no hook, no filter, no log entry, and no plugin surface where it could appear. It fails at the first of [the links in the chain that ends in an AI answer](/ai-crawlers-request-by-request/).

This is a general limit and worth being clear about, including for our own product. RankReady, our WordPress AI SEO plugin, controls what WordPress declares: robots directives, crawler rules, Markdown endpoints, schema, freshness signals. Five minutes to set up, runs alongside Rank Math, Yoast, AIOSEO or SEOPress. What it cannot do, and what no plugin can do, is override a decision made above the application. Anyone claiming their plugin guarantees AI crawler access has not understood where the block lives. It is on [the WordPress.org plugin directory](https://wordpress.org/plugins/rankready-ai-llm-seo/) if you want the declarative half handled properly.

The declaration and the behaviour are two different things, and only one of them is under your control from inside the dashboard. That gap is the same one that makes the whole audit-tool category unreliable, which is covered in [the fourteen free checkers we read](/ai-readiness-checkers/).

## Test your own site in four commands

These run on any Mac or Linux terminal, and in Windows through WSL. Replace the domain with yours. Nothing here changes anything on your site, and a shorter [five command reachability test](/ai-crawler-test/) covers the same ground if you want it condensed.

**One: request your homepage as GPTBot.**

`curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" \
https://example.com/`

**Two: request the same page as an ordinary browser, for comparison.**

`curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
https://example.com/`

**Three: request a deep post as GPTBot, not the homepage.**

`curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" \
https://example.com/your-best-post/`

**Four: request robots.txt as GPTBot.**

`curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" \
https://example.com/robots.txt`

Run each one three times. A single request can lie, because rate limiters have thresholds and caches have warm and cold states. Three consistent results is a finding. One result is a coincidence.

## Reading what comes back

| Command 1 (GPTBot page) | Command 2 (browser) | Command 4 (robots.txt) | What it means |
| ----------------------- | ------------------- | ---------------------- | ------------- |
| 200 | 200 | 200 | Clean. Crawler is being served normally. |
| 429 | 200 | 200 | The pattern above. A rate limit applied to the crawler only. |
| 403 | 200 | 200 | An outright block on user agent, usually a security plugin or firewall rule. |
| 200 | 200 | 404 | No robots.txt at all, which is a separate and smaller problem. |
| 503 | 503 | 503 | Site-wide issue. Not a crawler problem. |

The row that matters is the second one. Command 1 failing while command 2 succeeds means the difference is your user agent, and the only thing that reads your user agent before WordPress is your host, your firewall or your CDN.

Worth repeating the same four commands for other agents by swapping the user agent string. ClaudeBot, PerplexityBot, Google-Extended, Bingbot and CCBot are all worth checking.

## Three other places the block hides

The firewall rule is the hardest one to find. Three others are common enough to check while you are already in there.

**The security plugin.** Bot protection features in WordPress security plugins often ship with a bot list, and AI crawlers get added to those lists faster than most users notice. This one does live inside WordPress, so it is findable, usually under a heading like bad bot blocking or user agent filtering.

**The CDN or edge network.** Bot management sits in front of your origin by definition, and several providers now offer AI crawler control as a one-click product feature. Someone in your organisation may have clicked it, in a dashboard nobody else logs into, or nobody clicked anything at all and [Cloudflare's new crawler default](/cloudflare-crawler-default-wordpress/) decided for you.

**The robots.txt that says the wrong thing.** Group matching in robots.txt is more subtle than most people assume, and a rule intended for one agent can end up governing another because the most specific matching group wins outright rather than combining with the general one. A file can read as permissive to a human and parse as restrictive to a crawler.

The four commands catch all three, which is the reason to test by behaviour rather than by reading configuration. You do not have to know which layer did it to know that it happened.

## Getting it fixed

Once you have three consistent 429s, the conversation with your host is short, and it works far better with evidence attached.

- **Send the exact commands and the exact output.** Paste all three runs. Hosting support treats a reproducible curl transcript very differently from a description of a problem.

- **Name the layer.** Ask specifically whether a WAF rule, a bot management ruleset, or a rate limit is matching on user agent. Those are the three places it lives.

- **Ask for an allowlist rather than a rule removal.** You probably do want rate limiting in general. What you want is the named AI crawlers exempted from the aggressive tier.

- **Ask whether it is a platform default.** Several hosts and edge providers ship AI crawler blocking on by default now, sometimes as a selling point. If it is a default, it is a toggle, and you can decide deliberately rather than discovering it by accident.

- **Re-test after the change.** Same four commands, three runs each. Do not accept "it should be fine now" without the status codes.

Point 4 deserves emphasis. Blocking AI crawlers is a legitimate choice, and plenty of publishers make it on purpose. The problem in this case was not the policy. The problem was that nobody had chosen it, nobody knew it was on, and the site's own robots.txt was publishing the opposite.

## What it actually costs

A 429 to GPTBot means OpenAI's crawler does not retrieve your pages, and [what GPTBot is for, against its two sibling agents](/gptbot-wordpress/), decides how much that matters. That is a crawl fact, and it is certain. What follows from it is less certain, and anyone who tells you exactly what it costs in citations is guessing.

What can be said: content that is never fetched cannot be used. Every other piece of work in this field assumes retrieval as its base case. Sentence-level editing assumes retrieval. Schema assumes retrieval. Freshness assumes retrieval. All of it sits on top of a request that has to succeed.

There is also a compounding effect worth noting. In the analysis of 153,425 citations across six platforms in May 2026, stale pages were around three times more likely to lose citations they previously held. A crawler that has been backing off for months is not seeing your updates, so a page that was fresh at the last successful fetch is now, from that crawler's position, unchanged and ageing.

Check the crawl before spending a weekend on anything else. The sentence-level work in [the study that found no cited sentence above 18 words](/18-word-ceiling/) is genuinely the highest-yield editorial change available, and it is worth precisely nothing on a site returning 429.

## Why this case is probably common

Three things about this incident suggest it is not unusual.

The rule was invisible from every direction the site owner could look. The dashboard showed nothing. The pages loaded in a browser. Analytics were unaffected, because a crawler that never receives a page never fires a tracking script.

The exemption for robots.txt and the sitemap meant every automated check reported success. Any tool that reads the declaration rather than testing the behaviour would have passed this site, and most tools read the declaration.

And the site was on a mainstream stack with a mainstream host, running a normal WordPress install. Nothing exotic. If a rule like this can sit undetected on a site like that, the base rate is not one in a thousand.

## The habit worth forming

Four commands, once a quarter, on your top three URLs, for four or five user agents. Ten minutes total.

Hosting configurations change without announcement. Security vendors update their default rulesets. A migration moves you to a new stack with different defaults. None of those events send you an email, and none of them show up in WordPress.

The general lesson from this case is about trusting declarations over behaviour. Your robots.txt is a statement of intent. Your status code is what actually happened. When those two disagree, the status code is the only one that counts, and it is the one almost nobody checks.

When did you last request one of your own pages as a crawler and read the number that came back?