--- title: "The Five Layer AI SEO Audit, Run by Hand With curl" url: https://hostmy.blog/five-layer-ai-seo-audit/ date: 2026-09-13 modified: 2026-09-03 lang: en author: "Aditya Sharma" description: "Five layers in dependency order, each with a command you can run today. Fix layer one before you touch layer four, or you waste the work." categories: - "RankReady" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1922-1024x538.jpg word_count: 1643 --- # The Five Layer AI SEO Audit, Run by Hand With curl Most AI SEO audits start at the wrong end. Someone rewrites their sentences, adds schema, generates an llms.txt, then discovers three weeks later that the site was returning 403 to every AI crawler the whole time. The layers have a dependency order, and the order is not negotiable. Nothing above layer one matters if layer one fails. Below is the audit run by hand, five layers, each with a command. Roughly forty minutes for one site the first time, ten minutes on repeat. If forty minutes is more than you have, [the four-check short version of the same sweep](/ai-seo-checker-curl/) covers the essentials. ## Layer 1: can the crawler fetch the page at all Everything else depends on this. Start here every single time. `for ua in "GPTBot" "ClaudeBot" "PerplexityBot" "ChatGPT-User" "OAI-SearchBot"; do code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://yoursite.com/your-post/) printf '%-16s %s\n' "$ua" "$code" done` Those five all send a real HTTP user agent, so they can be tested this way. `Google-Extended` and `Applebot-Extended` cannot: they are robots.txt control tokens with no user agent of their own, so sending one as a header tests nothing and grepping a log for it always returns zero. You want `200` on every line you intend to allow. A `403` means a WAF, a security plugin or a bot rule is refusing the fetch, and no amount of content work reaches past that. Then read what your robots.txt actually says: `curl -s https://yoursite.com/robots.txt` Read it as a machine would, not as a human would. robots.txt matching is most-specific-wins rather than additive. A crawler that has its own named group ignores every rule in the wildcard group. Give `GPTBot` its own two line block and you have silently dropped all your `Disallow` rules for it. That trap has a full write up in [writing robots.txt rules that mean what you intended](/block-ai-crawlers-robots-txt/), and it is the single most common self-inflicted wound in this whole audit. Finally, confirm that requests are arriving at all: `grep -icE 'GPTBot|ClaudeBot|PerplexityBot|CCBot|ChatGPT-User' /var/log/nginx/access.log` Zero on a site with traffic means either they are being blocked upstream or your log format is dropping the user agent. Both are worth knowing. Three things break this layer more often than anything else. Cloudflare's bot protection at a setting nobody remembers choosing. A security plugin with an AI crawler blocklist enabled by default. And a hosting level rule applied at the account, not the site, so it survives every change you make inside WordPress. Test from outside your own network as well. A rule that whitelists your office IP will hand you a clean `200` while every crawler gets a challenge page. [Five curl commands for proving crawler access](/ai-crawler-test/) covers that check on its own. ## Layer 2: what a machine actually receives A `200` is not the same as a useful response. Measure the payload. `curl -s https://yoursite.com/your-post/ | wc -c` Then compare that against the visible text: `curl -s https://yoursite.com/your-post/ \ | sed -e 's/]*>.*<\/script>//g' -e 's/<[^>]*>//g' \ | tr -s ' \n' ' \n' | wc -c` The ratio between those two numbers is what you are looking at. A page here dropped from 4.7 MB of HTML to 18 KB of Markdown carrying the same words, roughly 260 times smaller. That gap is navigation, inline scripts, base64 images and markup. Check whether you serve a machine readable copy: `curl -sI -H "Accept: text/markdown" https://yoursite.com/your-post/ | grep -i 'content-type' curl -sI https://yoursite.com/your-post/index.md | grep -i -E 'http/|content-type'` The header route is honoured by coding agents only: Claude Code, Copilot Chat and CLI, Cursor, Microsoft Copilot, OpenClaw and OpenCode. ChatGPT browse, Claude.ai, Perplexity, Gemini and Grok do not send that header. So the distinct `.md` URL and the alternate link tag carry the reach, and the header alone does not. The mechanics are in [a lighter representation of the same page](/markdown-version-of-blog-posts/). Check the llms.txt while you are here, and check its type rather than its contents: `curl -sI https://yoursite.com/llms.txt | grep -i -E 'http/|content-type'` A `text/html` response means the file is unparseable to a strict client, which is a real failure found on real sites. Details in [generating an llms.txt on WordPress](/llms-txt-generator-wordpress/). ## Layer 3: structure Two things get measured here: heading order and answer position. Extract the heading skeleton: `curl -s https://yoursite.com/your-post/ \ | grep -o ']*>' \ | grep -o 'h[1-6]'` Read the sequence top to bottom. You want H1, then H2, then H3 under an H2, with no skipped levels. Sequential heading order correlates with a 2.8x lift. Then count them: `curl -s https://yoursite.com/your-post/ | grep -c ']*>'` More is not better. A page split into twenty sections is worse positioned than the same page with eight. That applies to FAQ entries too, since each one generates a heading, so cap them rather than maximising them. Answer position is the other half. 41.9 percent of AI citations come from the first 30 percent of a page. Open your post and ask where the direct answer sits. If it appears after the anecdote, the context and the caveats, it sits in the part of the page that rarely gets quoted. ## Layer 4: sentence length This is the layer with the hardest number attached to it. Across 11,346 cited sentences a study could extract, the mean cited sentence ran 9.27 words. The 6 to 10 word band carried 45.2 percent of all citations. Nothing longer than 18 words was cited once. Run your own page against that ceiling: `curl -s https://yoursite.com/your-post/ -o /tmp/p.html python3 - <<'PY' import re html = open('/tmp/p.html').read() body = re.sub(r'<(script|style)[^>]*>.*?', ' ', html, flags=re.S|re.I) text = re.sub(r'<[^>]+>', ' ', body) text = re.sub(r'\s+', ' ', text) sents = [s.strip() for s in re.split(r'(?<=[.!?])\s+', text) if len(s.split()) > 2] over = [s for s in sents if len(s.split()) > 18] print(f'{len(sents)} sentences, {len(over)} over 18 words ({len(over)*100//max(len(sents),1)}%)') for s in over[:10]: print(len(s.split()), s[:110]) PY` The percentage is your score. Anything over 40% means most of your page sits above the ceiling. The rewriting method, with before and after pairs, is in [writing sentences an AI can quote](/write-sentences-ai-can-quote/). ## Layer 5: freshness The median cited page is 298 days old. Old pages get quoted. Neglected pages lose what they had, at roughly three times the rate. Check what your pages claim about themselves: `curl -s https://yoursite.com/your-post/ | grep -o '"dateModified":"[^"]*"'` An absent `dateModified` is a gap. A `dateModified` identical to `datePublished` on a three year old post is a signal you probably do not want to be sending. Then check the site wide picture: `curl -s https://yoursite.com/post-sitemap.xml \ | grep -o '[^<]*' \ | sed 's/<[^>]*>//g' | sort | uniq -c | sort -rn | head` If every page shares one identical timestamp, something is rewriting `lastmod` on every build and the field is meaningless. What is worth updating and what should be left alone is covered in [freshness without the pointless date bump](/content-freshness-ai-answers/). ## What this audit cannot tell you Worth being straight about the limits, because the feedback loop here is genuinely bad. This audit measures whether machines can reach your pages, parse them, and find quotable units inside them. Every one of those is verifiable with the commands above, and every one of them either passes or fails. What it does not measure is whether you get cited. No rank tracker exists for this. Most AI clients send no referrer, so your analytics will not tell you either. Anyone offering you a citation score is modelling, not measuring, and the honest position is that crawlability is the part you control. Which parts of this field are sold on that gap is the subject of [an honest accounting of the AI SEO pitch](/what-ai-seo-means/). That is also why the order matters so much. You cannot A/B test your way to the answer, so you fix the things that are provably broken and stop there. ## The order matters more than the checklist | Layer | Question | One command | | ----- | -------- | ----------- | | 1 Fetch | Does the crawler get a 200 | `curl -A "GPTBot" -o /dev/null -w '%{http_code}'` | | 2 Payload | What does it actually receive | `curl -s URL | wc -c` | | 3 Structure | Are headings sequential and capped | `grep -o '