---
title: "Four curl Checks for Whether a Page Is Easy to Fetch and Read"
url: https://hostmy.blog/ai-seo-checker-curl/
date: 2026-09-18
modified: 2026-11-05
lang: en
author: "Aditya Sharma"
description: "Four curl checks that show whether a page is easy to fetch and read: reachability, rules, payload and sentence shape."
categories:
  - "AI Search"
image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1877-1024x538.jpg
word_count: 1541
---

# Four curl Checks for Whether a Page Is Easy to Fetch and Read

Four checks tell you whether a page is easy for an automated reader to fetch and read. Reachability, rules, payload, and whether a short claim sentence sits near the top. Each of the four is a single command with a clear pass condition.

Most AI SEO checker tools score the fourth one and assume the first three. That is backwards. A page can be beautifully structured and still be refused at the door, and no amount of rewriting fixes a 403.

Run these in order. Stop at the first failure, because the later checks are meaningless until the earlier ones pass. [The full five layer sweep](/five-layer-ai-seo-audit/) adds freshness and a site-wide pass on top of these four.

| Check | Command tests | Pass condition |
| ----- | ------------- | -------------- |
| 1. Reachability | Status code by user agent | 200 for every agent you allow |
| 2. Rules | robots.txt group matching | The group you think applies actually applies |
| 3. Payload | Bytes returned and content type | Under a megabyte, Markdown offered |
| 4. Sentence shape | Sentence length and position | Short claim sentence in the first third (our own rule of thumb) |

Four checks, in the order that makes the later ones meaningful

Four checks, in the order that makes the later ones meaningful

Reachable
curl -A GPTBot

Right robots
group
most specific wins

Small enough to
read
HTML against Markdown

Short claim sentence
early
first third

200

Allow

18 KB
A browser getting a 200 proves nothing, because your browser is not the user agent being
refused.

Check one and two decide whether three and four are measuring anything. Running them out of order is how a site with a 403 gets a clean schema report.

## Check one: your browser getting a 200 proves nothing

Your browser gets a 200. That tells you almost nothing, because your browser is not what fetches the page when somebody asks a question.

`for UA in "Mozilla/5.0" "GPTBot/1.4" "PerplexityBot/1.0" "ClaudeBot/1.0"; do
printf '%-20s ' "$UA"
curl -sS -o /dev/null -w '%{http_code}\n' -A "$UA" https://example.com/your-post/
done`

Pass condition: every agent you intend to allow returns 200. A 403 or a 429 for one agent while your browser gets 200 is a server-side or CDN rule, not a content problem.

Measured on our own infrastructure, exactly this pattern showed up: a site returned 429 to GPTBot while its robots.txt said Allow. The refusal reproduced 3 times out of 3. Static assets were refused too, while robots.txt and the sitemap both returned 200, which placed the block at the web server layer rather than inside WordPress.

That signature is worth memorising. robots.txt fine, sitemap fine, HTML refused, images refused. When you see it, stop reading plugin settings and go look at the rate limiter.

One 429 can be a burst limit doing its job. Three in a row against the same URL is a rule, and [a standalone crawler reachability test](/ai-crawler-test/) is the fastest way to establish which of the two you have.

## Check two: the robots.txt group you think applies is the one that applies

robots.txt matching is most-specific-wins, not additive. A named crawler group replaces the wildcard group entirely.

That single rule causes more accidental exposure and more accidental blocking than any other line in the file. A site with twenty disallow lines under `User-agent: *` and two lines under `User-agent: GPTBot` has not added two rules for GPTBot. It has replaced all twenty with those two.

`curl -sS https://example.com/robots.txt`

Read the output and answer one question for each named agent: if this agent only obeys the lines in its own group, is that group complete? If the answer is no, the group needs the wildcard rules copied into it, or the group should not exist.

Cloudflare has also changed its defaults: since 15 September 2026, newly onboarded domains block Training and Agent crawlers on ad-bearing pages, and multi-purpose crawlers are judged on all of their behaviours. If your site sits behind Cloudflare, the decision may be made before the request reaches your origin, and your access log will show nothing at all. The details are in [what the Cloudflare crawler change means for WordPress blogs](/cloudflare-crawler-default-wordpress/).

## Check three: what comes back is small enough to be worth reading

Measured on our own infrastructure, a single page returned 4.7 MB of HTML against 18 KB of the same content in Markdown. Roughly 260 times the payload for the same words.

`curl -sS -o /dev/null -w 'html %{size_download} bytes\n' https://example.com/your-post/
curl -sS -H 'Accept: text/markdown' -o /dev/null \
-w 'md %{size_download} bytes %{content_type}\n' https://example.com/your-post/`

Pass condition on the first line: the HTML is under a megabyte. Anything multi-megabyte is mostly scripts, embeds and a related-posts grid. Running that byte count across every URL gives you [a site-wide sweep for pages too heavy to read](/find-pages-ai-cannot-parse/).

Pass condition on the second line: the content type comes back as `text/markdown` and the byte count drops sharply. If both lines return the same size and the same `text/html`, no negotiation is happening.

Worth knowing before you get excited about that second line: Markdown content negotiation is honoured by coding agents only. Claude Code, Copilot Chat and CLI, Cursor, Microsoft Copilot, OpenClaw and OpenCode ask for it. ChatGPT browse, Claude.ai, Perplexity, Gemini and Grok do not. So the Markdown endpoint helps a real and growing set of clients, and it does nothing for the five chat assistants people name first. Anyone selling it as the fix for AI visibility is overselling it.

While you are here, check the structured data comes back in the raw HTML rather than being injected by JavaScript, which is the first step in [checking schema without the Rich Results Test](/check-schema-without-google/).

`curl -sS https://example.com/your-post/ | grep -o 'application/ld+json' | wc -l`

Zero means either no schema or schema that only exists after JavaScript runs. Which types are worth having, and which are theatre, is covered in [what each schema type does](/schema-markup-guide-ai/).

## Check four: a short claim sentence sits in the first third

This is the check every AI SEO checker tool runs, and it is genuinely the one that matters once the first three pass.

In one self-published study of Gemini citations (Daniel Shashko, May 2026), the 11,346 extracted sentences had a mean of 9.27 words, a median of 10 and a longest of 18. The 6 to 10 word band carried 45.2 percent.

Position matters for readers too. In that sample about 42 percent of the matched sentences came from the first 30 percent of the source page, with the mean sitting 37 percent down, across the 9,968 placed sentences.

So the test is mechanical. Extract the sentences from the top third of your page and count their words.

`curl -sS https://example.com/your-post/ \
| sed -e 's/<[^>]*>//g' \
| tr '.' '\n' \
| awk 'NF>2 {print NF"\t"$0}' \
| head -40`

The first column is a word count. Scan the first forty sentences for any row under 18. If every number in that column is in the twenties and thirties, you have written a page with no short claim in the part of it that readers see first.

Structure is worth a look too. Keep headings sequential, meaning H2 followed by H3 without skipping levels. More headings is not automatically better. Order your headings properly, and stop adding them past the point where they describe real sections.

Readability splits oddly in the same data. Of 11,215 scored Gemini sentences, 22.9 percent scored very easy, 20.5 percent scored very confusing, and only 2.6 percent sat in the Flesch 50 to 59 band. The median was 66.4. Plain writing and dense technical writing were both well represented in that sample. The magazine middle was not.

## What this checks and what it cannot check

These four commands verify crawlability: whether an agent can reach your page, whether your rules say what you think, whether the response is light enough to read, and whether it contains something short enough to lift.

None of them predicts whether you will be cited. Nobody can promise that, and the platform numbers show why. In one self-published study, only 23.05 percent of cited URLs ranked in the organic top ten. For ChatGPT the figure was 4.2 percent. AI Mode and Gemini, both Google products, shared just 4.66 percent of cited domains. The selection layer is opaque and it differs by platform.

What you control is the four gates. Failing one of them guarantees nothing downstream happens.

## Running these checks continuously

The origin-side half of this checklist, robots.txt with Content Signals, a Markdown endpoint, schema in the raw HTML, an llms.txt file, is what RankReady sets up in about five minutes on one settings screen. It sits alongside Rank Math, Yoast, AIOSEO or SEOPress and adds no sitemap of its own.

Check one is the exception. A refusal at the web server or CDN layer sits above WordPress entirely, and no plugin reaches that far, worth saying plainly rather than pretending otherwise. The plugin is on [the WordPress.org directory](https://wordpress.org/plugins/rankready-ai-llm-seo/), and the full crawler-chain background is in [what happens between an AI answer and your server](/ai-crawlers-request-by-request/).

## One question before you optimise anything

Pick your three best-performing posts and run check one against each of them.

If all three return 200 for every agent, your problem is genuinely a content problem and check four is where to spend the afternoon. If even one returns a 403 or a 429, what have you been optimising for?