--- title: "What Happens Between an AI Answer and Your Server, Request by Request" url: https://hostmy.blog/ai-crawlers-request-by-request/ date: 2026-09-18 modified: 2026-09-03 lang: en author: "Aditya Sharma" description: "An AI answer that cites you is the end of a chain of ordinary HTTP requests. Here is every link in that chain." categories: - "AI Search" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1876-1024x538.jpg word_count: 1581 --- # What Happens Between an AI Answer and Your Server, Request by Request An AI answer that quotes your page is the end of a chain of ordinary HTTP requests. Every link in that chain is written down in your access log. Nothing in it is magic and nothing in it is hidden from you. The chain has five links: a robots.txt fetch, a decision at your web server, a page fetch, a response body, and a sentence selected out of that body. A failure at any link ends the chain silently. Your analytics will never mention it, because a crawler does not run JavaScript. Two of those links are usually the ones that break, and neither is a content problem. ## The visitors split into two kinds that behave nothing alike Bulk crawlers fetch broadly and on their own schedule. Live retrieval agents fetch one or two URLs because somebody asked a question thirty seconds ago. The distinction matters because they are often governed by different rules and blocked by different mistakes. A bulk crawler that is refused costs you future coverage. A retrieval agent that is refused costs you an answer that was already in progress. The second kind is the one worth watching in your log. Each of those requests maps to a real person waiting on a real answer. A bulk crawl tells you a machine has your site on a list. A retrieval fetch tells you a question was asked and your page was a candidate for it. Separating the two is a matter of [pulling crawler hits out of a raw access log](/read-server-log-ai-crawlers/), which needs no plugin at all. | Behaviour | Fetch pattern | What a block costs you | | --------- | ------------- | ---------------------- | | Bulk crawl | Many URLs, own schedule, repeated | Future coverage of the whole site | | Live retrieval | One or two URLs, triggered by a question | An answer already being written | Operator user agent strings change, and a stale rule is worse than no rule. Check each operator's own published documentation before you write anything into robots.txt, rather than copying a list out of a blog post. Question to cited sentence, one request at a time Question to cited sentence, one request at a time robots.txt fetch /robots.txt Server decision allow or refuse Page fetch GET /post/ Response body 4.7 MB of HTML Sentence selection 9.27 words, mean 200 200 200 200 403 or 429 here nothing downstream runs 41.9 percent of citations were lifted from the first 30 percent of the body. Every stage after the server decision is conditional on it. A refusal at box two makes the remaining three unmeasurable rather than merely worse. ## Your server can refuse a crawler while robots.txt says allow This is the failure that hides best, because the file everybody checks is the file that is fine. Measured on our own infrastructure: a site returned 429 to GPTBot while its robots.txt said Allow. The refusal reproduced 3 times out of 3. The block sat at the web server layer, not in WordPress, because static assets were refused too while robots.txt and the sitemap both returned 200. Read that combination again, because it is the diagnostic signature. robots.txt returns 200. The sitemap returns 200. The HTML returns 429 or 403. That pattern means a rate limiter or a bot rule above WordPress is making the decision, and no plugin can override it. Plenty of shared platforms do exactly this, and [hosts that refuse AI crawlers without saying so](/host-blocking-ai-crawlers/) rarely mention it in support docs. `curl -sS -o /dev/null -w '%{http_code}\n' \ -A 'Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)' \ https://example.com/your-post/` Run that three times. One 429 can be a coincidence. Three is a rule, and [five curl commands that show whether crawlers can reach you](/ai-crawler-test/) will tell you how far the refusal spreads across the rest of the site. ## robots.txt is matched most specific wins, not additively A named crawler group replaces the wildcard group entirely. It does not add to it. So a site with a long `User-agent: *` block and a short `User-agent: GPTBot` block underneath has not tightened anything for GPTBot. It has replaced every wildcard rule with the two lines in the named group. Every disallow in the wildcard block stops applying to that agent. This one bites in both directions. People who meant to block end up allowing, and people who meant to allow one path end up opening everything. Settle [whether GPTBot gets an allow rule](/gptbot-wordpress/) before you write the group, not after. ## Cloudflare moved the default, so the decision may not be yours any more On 15 September 2026 Cloudflare changed its defaults: for domains newly onboarding, Training and Agent crawlers are blocked on pages that display ads, while Search stays allowed. The same change evaluates multi-purpose crawlers on all of their behaviours, so blocking Training now also blocks Googlebot, Applebot and BingBot for anyone who chose that. If your site sits behind Cloudflare, the refusal can happen before the request reaches your origin. Your log will show nothing, because the request never arrived. That is worth checking before you spend an afternoon reading nginx config. More on the specifics in [what the Cloudflare change means for WordPress](/cloudflare-crawler-default-wordpress/). ## The response body decides the cost of reading you Measured on our own infrastructure: one page returned 4.7 MB of HTML against 18 KB of the same content in Markdown. That is roughly 260 times the payload for the same words. A retrieval agent working through a 4.7 MB document is spending its budget on your navigation, your consent script and your related-posts grid. The sentence it needs is somewhere inside that. Markdown content negotiation is honoured by coding agents only. Claude Code, Copilot Chat and CLI, Cursor, Microsoft Copilot, OpenClaw and OpenCode will ask for and accept a Markdown response. ChatGPT browse, Claude.ai, Perplexity, Gemini and Grok do not. `curl -sSI -H 'Accept: text/markdown' https://example.com/your-post/ | grep -i content-type` A `text/markdown` content type back means negotiation is working. `text/html` means you are serving the heavy version to everybody. Serving [a Markdown copy of a post](/markdown-version-of-blog-posts/) without splitting your content into two sources of truth is its own small job. ## Selection happens inside the first third of the page Once the body arrives, a sentence gets picked out of it. Across 9,968 cited sentences a study could place, 41.9 percent came from the first 30 percent of the page. Mean cited-sentence position across those 9,968 cited sentences was 37 percent down. The cited sentences themselves are short. Mean length was 9.27 words, median 10, and no sentence longer than 18 words was cited once in the whole set. The 6 to 10 word band alone carried 45.2 percent of citations. Front-load the claim, then keep the claim sentence short enough to lift. The rest of the page can breathe. ## The six platforms do not fetch at anything like the same rate Citation volume across those 153,425 records was heavily lopsided. AI Mode accounted for 88,392 citations, Grok 30,676, Gemini 13,487, Copilot 8,779, Perplexity 8,562 and ChatGPT 3,529. The per-query behaviour differs even more than the totals. Grok returned 35.79 citations per query. Gemini returned 7.06. A platform that cites thirty-five sources per answer is reading widely and shallowly. A platform that cites seven is being selective about which pages it opens at all. There is also very little agreement between them. AI Mode and Gemini, both Google products, shared only 4.66 percent of cited domains. Only 23.05 percent of cited URLs ranked in the organic top ten at all, and for ChatGPT that figure was 4.2 percent. Perplexity was the outlier with 50.3 percent overlap with organic top-10. Two practical consequences follow. Being fetchable matters separately from ranking, because most cited URLs were not top-ten pages. And checking one assistant tells you very little about the other five. ## Trace your own chain in ten minutes Four commands, in order, against a page you care about. `curl -sS https://example.com/robots.txt | head -40 curl -sS -o /dev/null -w '%{http_code} %{size_download}\n' https://example.com/your-post/ curl -sS -o /dev/null -w '%{http_code}\n' -A 'GPTBot' https://example.com/your-post/ curl -sSI -H 'Accept: text/markdown' https://example.com/your-post/ | head -20` Line one shows the rules. Line two shows the normal response and its weight in bytes. Line three shows whether an agent gets the same treatment you do. Line four shows whether a lighter version exists. A fuller version of this sequence, with the pass and fail conditions written out, is in [the four things an assistant checks](/ai-seo-checker-curl/). ## What RankReady does with this chain RankReady is a WordPress AI SEO plugin that works on the links you actually control: the robots.txt rules including Content Signals, a Markdown endpoint for the agents that ask for one, schema, and an llms.txt file if you want one. Setup takes about five minutes. It runs alongside Rank Math, Yoast, AIOSEO and SEOPress rather than replacing them. Your existing plugin keeps owning titles, meta descriptions and sitemaps. RankReady adds the crawler-facing layer on top. What it cannot do is fix a refusal that happens above WordPress. A 429 from your web server or your CDN is a server config job, and any plugin that claims otherwise is claiming something it cannot deliver. The honest scope here is crawlability: making sure a request that reaches WordPress gets a clean, light, well-marked answer. You can install it from [the WordPress.org plugin directory](https://wordpress.org/plugins/rankready-ai-llm-seo/). ## The question worth asking first Run the third command above against your five best posts. If any of them return something other than 200, the content question is not the one in front of you. So which link in your chain have you actually checked, and which one have you assumed?