--- title: "The Four Things an Assistant Checks Before It Uses You as a Source" url: https://hostmy.blog/ai-seo-checker-curl/ date: 2026-09-18 modified: 2026-09-03 lang: en author: "Aditya Sharma" description: "Four checks stand between your page and an assistant quoting it. Each one is a curl command with a pass condition." categories: - "AI Search" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1877-1024x538.jpg word_count: 1521 --- # The Four Things an Assistant Checks Before It Uses You as a Source Four checks stand between your page and an assistant using it as a source. Reachability, rules, payload, and whether a liftable sentence exists near the top. Each of the four is a single command with a clear pass condition. Most AI SEO checker tools score the fourth one and assume the first three. That is backwards. A page can be beautifully structured and still be refused at the door, and no amount of rewriting fixes a 403. Run these in order. Stop at the first failure, because the later checks are meaningless until the earlier ones pass. [The full five layer sweep](/five-layer-ai-seo-audit/) adds freshness and a site-wide pass on top of these four. | Check | Command tests | Pass condition | | ----- | ------------- | -------------- | | 1. Reachability | Status code by user agent | 200 for every agent you allow | | 2. Rules | robots.txt group matching | The group you think applies actually applies | | 3. Payload | Bytes returned and content type | Under a megabyte, Markdown offered | | 4. Liftability | Sentence length and position | Claim under 18 words in the first third | Four checks, in the order that makes the later ones meaningful Four checks, in the order that makes the later ones meaningful Reachable curl -A GPTBot Right robots group most specific wins Small enough to read HTML against Markdown Quotable sentence early first third 200 Allow 18 KB A browser getting a 200 proves nothing, because your browser is not the user agent being refused. Check one and two decide whether three and four are measuring anything. Running them out of order is how a site with a 403 gets a clean schema report. ## Check one: your browser getting a 200 proves nothing Your browser gets a 200. That tells you almost nothing, because your browser is not what fetches the page when somebody asks a question. `for UA in "Mozilla/5.0" "GPTBot/1.4" "PerplexityBot/1.0" "ClaudeBot/1.0"; do printf '%-20s ' "$UA" curl -sS -o /dev/null -w '%{http_code}\n' -A "$UA" https://example.com/your-post/ done` Pass condition: every agent you intend to allow returns 200. A 403 or a 429 for one agent while your browser gets 200 is a server-side or CDN rule, not a content problem. Measured on our own infrastructure, exactly this pattern showed up: a site returned 429 to GPTBot while its robots.txt said Allow. The refusal reproduced 3 times out of 3. Static assets were refused too, while robots.txt and the sitemap both returned 200, which placed the block at the web server layer rather than inside WordPress. That signature is worth memorising. robots.txt fine, sitemap fine, HTML refused, images refused. When you see it, stop reading plugin settings and go look at the rate limiter. One 429 can be a burst limit doing its job. Three in a row against the same URL is a rule, and [a standalone crawler reachability test](/ai-crawler-test/) is the fastest way to establish which of the two you have. ## Check two: the robots.txt group you think applies is the one that applies robots.txt matching is most-specific-wins, not additive. A named crawler group replaces the wildcard group entirely. That single rule causes more accidental exposure and more accidental blocking than any other line in the file. A site with twenty disallow lines under `User-agent: *` and two lines under `User-agent: GPTBot` has not added two rules for GPTBot. It has replaced all twenty with those two. `curl -sS https://example.com/robots.txt` Read the output and answer one question for each named agent: if this agent only obeys the lines in its own group, is that group complete? If the answer is no, the group needs the wildcard rules copied into it, or the group should not exist. Cloudflare has also changed its defaults: since 15 September 2026, newly onboarded domains block Training and Agent crawlers on ad-bearing pages, and multi-purpose crawlers are judged on all of their behaviours. If your site sits behind Cloudflare, the decision may be made before the request reaches your origin, and your access log will show nothing at all. The details are in [what the Cloudflare crawler change means for WordPress blogs](/cloudflare-crawler-default-wordpress/). ## Check three: what comes back is small enough to be worth reading Measured on our own infrastructure, a single page returned 4.7 MB of HTML against 18 KB of the same content in Markdown. Roughly 260 times the payload for the same words. `curl -sS -o /dev/null -w 'html %{size_download} bytes\n' https://example.com/your-post/ curl -sS -H 'Accept: text/markdown' -o /dev/null \ -w 'md %{size_download} bytes %{content_type}\n' https://example.com/your-post/` Pass condition on the first line: the HTML is under a megabyte. Anything multi-megabyte is mostly scripts, embeds and a related-posts grid. Running that byte count across every URL gives you [a site-wide sweep for pages too heavy to read](/find-pages-ai-cannot-parse/). Pass condition on the second line: the content type comes back as `text/markdown` and the byte count drops sharply. If both lines return the same size and the same `text/html`, no negotiation is happening. Worth knowing before you get excited about that second line: Markdown content negotiation is honoured by coding agents only. Claude Code, Copilot Chat and CLI, Cursor, Microsoft Copilot, OpenClaw and OpenCode ask for it. ChatGPT browse, Claude.ai, Perplexity, Gemini and Grok do not. So the Markdown endpoint helps a real and growing set of clients, and it does nothing for the five chat assistants people name first. Anyone selling it as the fix for AI visibility is overselling it. While you are here, check the structured data comes back in the raw HTML rather than being injected by JavaScript, which is the first step in [checking schema without the Rich Results Test](/check-schema-without-google/). `curl -sS https://example.com/your-post/ | grep -o 'application/ld+json' | wc -l` Zero means either no schema or schema that only exists after JavaScript runs. Which types are worth having, and which are theatre, is covered in [schema that survives being lifted](/schema-markup-guide-ai/). ## Check four: a quotable sentence exists in the first third This is the check every AI SEO checker tool runs, and it is genuinely the one that matters once the first three pass. Across 11,346 cited sentences a study could extract, the mean cited sentence ran 9.27 words, the median 10, and no sentence longer than 18 words was cited once. The 6 to 10 word band alone carried 45.2 percent of citations. Position matters nearly as much as length. 41.9 percent of citations came from the first 30 percent of the page, with the mean sitting 37 percent down, across the 9,968 cited sentences analysed. So the test is mechanical. Extract the sentences from the top third of your page and count their words. `curl -sS https://example.com/your-post/ \ | sed -e 's/<[^>]*>//g' \ | tr '.' '\n' \ | awk 'NF>2 {print NF"\t"$0}' \ | head -40` The first column is a word count. Scan the first forty sentences for any row under 18. If every number in that column is in the twenties and thirties, you have written a page with no liftable claim in the part of it that gets read. Structure carries measurable weight too. Sequential headings, meaning H2 followed by H3 without skipping levels, showed a 2.8x lift. More headings is not automatically better. Order your headings properly, and stop adding them past the point where they describe real sections. Readability splits oddly in the same data. 22.9 percent of cited pages scored very easy, 20.5 percent scored very confusing, and only 2.6 percent sat in the Flesch 50 to 59 middle. Median was 66.4. Plain writing is well represented and dense technical writing is well represented. The magazine middle is not. ## What this checks and what it cannot check These four commands verify crawlability: whether an agent can reach your page, whether your rules say what you think, whether the response is light enough to read, and whether it contains something short enough to lift. None of them predicts whether you will be cited. Nobody can promise that, and the platform numbers show why. Only 23.05 percent of cited URLs ranked in the organic top ten. For ChatGPT the figure was 4.2 percent. AI Mode and Gemini, both Google products, shared just 4.66 percent of cited domains. The selection layer is opaque and it differs by platform. What you control is the four gates. Failing one of them guarantees nothing downstream happens. ## Running these checks continuously The origin-side half of this checklist, robots.txt with Content Signals, a Markdown endpoint, schema in the raw HTML, an llms.txt file, is what RankReady sets up in about five minutes on one settings screen. It sits alongside Rank Math, Yoast, AIOSEO or SEOPress without touching your titles, meta descriptions or sitemap. Check one is the exception. A refusal at the web server or CDN layer sits above WordPress entirely, and no plugin reaches that far, worth saying plainly rather than pretending otherwise. The plugin is on [the WordPress.org directory](https://wordpress.org/plugins/rankready-ai-llm-seo/), and the full crawler-chain background is in [what happens between an AI answer and your server](/ai-crawlers-request-by-request/). ## One question before you optimise anything Pick your three best-performing posts and run check one against each of them. If all three return 200 for every agent, your problem is genuinely a content problem and check four is where to spend the afternoon. If even one returns a 403 or a 429, what have you been optimising for?