---
title: "Find Every Page on Your Site an AI Cannot Parse Cheaply"
url: https://hostmy.blog/find-pages-ai-cannot-parse/
date: 2026-09-21
modified: 2026-09-21
lang: en
author: "Aditya Sharma"
description: "A 200 status is not the same as a readable page. Here is a script that measures text-to-HTML ratio across your whole sitemap."
categories:
  - "AI Search"
image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1885-1024x538.jpg
word_count: 1619
---

# Find Every Page on Your Site an AI Cannot Parse Cheaply

A page can return a clean 200 and still be close to useless to a machine.

The measurement that matters is how much of what you send is actually content. Send 145 kilobytes of HTML wrapped around 2 kilobytes of text and a crawler spends almost all of its effort on your theme. [A Markdown copy served at its own URL](/markdown-version-of-blog-posts/) is one way out of that.

This post gives you a script that walks your sitemap and reports that ratio for every page, plus heading structure and Markdown availability. It runs on stock Python 3 with no packages to install.

One page, two representations, drawn to scale
One page, two representations, drawn to scale
Rendered HTML

4.7 MB
Same content as Markdown

18 KB
The difference is navigation, inline scripts, base64 images and markup.

The bar for the Markdown copy is not a rendering error. The same words came to 18 KB against 4.7 MB, roughly 260 times smaller.

## Step 1: what the ratio actually means

Take the raw bytes the server sends. Extract the visible text, throwing away scripts, styles and inline SVG. Divide.

A content page that lands in the low single digits is nearly all wrapper. A page in the forty to sixty percent band is mostly content. Neither number is a score anyone grades you on, and there is no published threshold to hit. It is a comparison tool: it tells you which of your own pages are the expensive ones.

Two things make the number worse than it needs to be. Inline JSON blobs from page builders and block editors, and script tags carrying data rather than code. Both are invisible to you and enormous to a parser, and both sit in the payload layer of [the audit that puts access before structure](/five-layer-ai-seo-audit/).

## Step 2: the script

Save this as `parsecost.py`.

`#!/usr/bin/env python3
"""How expensive is each page to parse? Reads a sitemap or a list of URLs."""
import re, sys, time, urllib.request, urllib.error
from html.parser import HTMLParser

SKIP = {"script", "style", "noscript", "svg", "template"}

class Page(HTMLParser):
def __init__(self):
super().__init__()
self.text, self.depth = [], 0
self.headings, self._h = [], None
def handle_starttag(self, tag, attrs):
t = tag.lower()
if t in SKIP:
self.depth += 1
if re.fullmatch(r"h[1-6]", t):
self._h = int(t[1])
self.headings.append([self._h, ""])
def handle_endtag(self, tag):
t = tag.lower()
if t in SKIP and self.depth:
self.depth -= 1
if re.fullmatch(r"h[1-6]", t):
self._h = None
def handle_data(self, data):
if self.depth:
return
if self._h is not None and self.headings:
self.headings[-1][1] += data
if data.strip():
self.text.append(data)
def body_text(self):
return re.sub(r"\s+", " ", " ".join(self.text)).strip()

def get(url, timeout=30):
req = urllib.request.Request(url, headers={
"User-Agent": "Mozilla/5.0 (compatible; parsecost/1.0)",
"Accept-Encoding": "identity"})
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.read()

def sitemap_urls(url, depth=0, max_depth=3, out=None):
"""Walk sitemap indexes, which are often nested more than one level."""
out = [] if out is None else out
if depth > max_depth or len(out) > 5000:
return out
body = get(url).decode("utf-8", "replace")
locs = re.findall(r"<loc>\s*([^<\s]+)\s*</loc>", body)
if "<sitemapindex" in body:
for child in locs:
try:
sitemap_urls(child, depth + 1, max_depth, out)
except Exception as e:
print(" skipped %s (%s)" % (child, e), file=sys.stderr)
if len(out) > 5000:
break
else:
out.extend(locs)
return out

def heading_breaks(headings):
"""Count skipped levels, for example an h2 followed by an h4."""
breaks, prev = 0, None
for level, _ in headings:
if prev is not None and level > prev + 1:
breaks += 1
prev = level
return breaks

def markdown_available(url):
req = urllib.request.Request(url, headers={
"User-Agent": "Mozilla/5.0 (compatible; parsecost/1.0)",
"Accept": "text/markdown"})
try:
with urllib.request.urlopen(req, timeout=20) as r:
return "markdown" in (r.headers.get("Content-Type") or "").lower()
except Exception:
return False

def main(src, limit):
if src == "-":
urls = [l.strip() for l in sys.stdin if l.strip()]
else:
try:
urls = sitemap_urls(src)
except urllib.error.HTTPError as e:
print("sitemap not found at %s (HTTP %d)." % (src, e.code))
print("Find the real one: curl -sSL %s/robots.txt | grep -i sitemap"
% "/".join(src.split("/")[:3]))
return 1
urls = urls[:limit]
if not urls:
print("no URLs found")
return 1
print("%-7s %-7s %-6s %-4s %-4s %-3s %s"
% ("html", "text", "ratio", "hdr", "brk", "md", "url"))
rows = []
for u in urls:
try:
raw = get(u)
except Exception as e:
print("error on %s: %s" % (u, e))
continue
p = Page()
p.feed(raw.decode("utf-8", "replace"))
text = p.body_text()
ratio = 100.0 * len(text) / max(1, len(raw))
rows.append((ratio, u))
print("%-7s %-7s %-6s %-4d %-4d %-3s %s"
% ("%dk" % (len(raw) // 1024), "%dk" % (len(text) // 1024),
"%.1f%%" % ratio, len(p.headings), heading_breaks(p.headings),
"yes" if markdown_available(u) else "no", u[:70]))
time.sleep(1.5)
rows.sort()
print("\nworst parse ratio first:")
for ratio, u in rows[:10]:
print(" %5.1f%% %s" % (ratio, u))
return 0

if __name__ == "__main__":
if len(sys.argv) < 2:
print(__doc__)
sys.exit(2)
lim = int(sys.argv[2]) if len(sys.argv) > 2 else 25
sys.exit(main(sys.argv[1], lim))`

## Step 3: run it

```
python3 parsecost.py https://yoursite.com/sitemap.xml 25
```

Real output from a run against wordpress.org, September 2026:

`html text ratio hdr brk md url
146k 2k 1.5% 4 0 yes https://wordpress.org/news/2003/04/welcome/
147k 2k 1.9% 4 0 yes https://wordpress.org/news/2003/04/comments-and-template/
147k 2k 1.7% 4 0 yes https://wordpress.org/news/2003/04/publicity-and-validation/
147k 2k 1.7% 4 0 yes https://wordpress.org/news/2003/04/the-7-release/

worst parse ratio first:
1.5% https://wordpress.org/news/2003/04/welcome/
1.7% https://wordpress.org/news/2003/04/publicity-and-validation/
1.7% https://wordpress.org/news/2003/04/the-7-release/
1.9% https://wordpress.org/news/2003/04/comments-and-template/`

Those are short archive posts, so 145 kilobytes of chrome around 2 kilobytes of text is the expected shape. The pattern to hunt for is a substantial article showing the same ratio, which means the wrapper has grown out of proportion to the writing.

For comparison, from the same run against three larger pages:

`264k 63k 23.8% 74 0 no https://yoast.com/wordpress-seo/
232k 37k 16.0% 64 2 yes https://wordpress.org/plugins/rankready-ai-llm-seo/
143k 4k 3.2% 5 0 yes https://wordpress.org/news/2026/08/open-weight/`

The script sleeps 1.5 seconds between requests. Leave that in. You are crawling your own site, but there is no reason to hammer it, and real crawlers pace themselves too, as [the round trip between a question and your access log](/ai-crawlers-request-by-request/) shows.

## Step 4: read the other three columns

The `hdr` column counts headings and `brk` counts skipped levels, an h2 followed by an h4 with no h3 between.

This is worth fixing because heading structure is one of the few things with a measured effect. Across a study of 153,425 citations from six platforms, sequential headings were associated with a 2.8 times lift. More headings is not automatically better. Sequential and few beats many and scattered, and structure is one of [the four things an assistant weighs before quoting you](/ai-seo-checker-curl/).

The `md` column says whether the page answers an `Accept: text/markdown` request with actual Markdown. Useful, with a caveat: that negotiation is honoured by coding agents, not by ChatGPT browse, Claude.ai, Perplexity, Gemini or Grok. Treat it as a bonus rather than the fix. The full check for it is in [testing whether AI crawlers can reach your site](/ai-crawler-test/).

## Step 5: check the front of the page, not just the whole page

Position matters more than most audits admit. In that same citation dataset, 41.9 percent of citations came from the first 30 percent of a page.

So run this on your worst offenders and look at where the substance starts:

`curl -sSL https://yoursite.com/your-post/ \
| python3 -c "import sys,re; from html.parser import HTMLParser
class P(HTMLParser):
d=0; out=[]
def handle_starttag(s,t,a):
if t in ('script','style','svg'): s.d+=1
def handle_endtag(s,t):
if t in ('script','style','svg') and s.d: s.d-=1
def handle_data(s,x):
if not s.d and x.strip(): s.out.append(x.strip())
p=P(); p.feed(sys.stdin.read()); print(' '.join(p.out)[:600])"`

That prints the first 600 characters of real text. Run against a live wordpress.org post it returns this:

`WordPress Signs the Open Weights and American AI Leadership Letter - WordPress
News Skip to content Showcase Plugins Themes Hosting News Resources Learn
WordPress Documentation Education Forums Developers Blocks Patterns Photos
Openverse WordPress.tv About About WordPress Make WordPress Events Five for
the Future Enterprise Gutenberg Job Board Swag Get WordPress Search in
WordPress.org Get WordPress News WordPress Signs the Open Weights and American
AI Leadership Letter All posts Search results Categories All Posts Awards
Community Design Development Documentation Education`

Six hundred characters in and the article has not started. That is the whole problem in one block: the title appears twice and everything between is navigation. If the answer to the page's own question is not in there, the reader who matters most has to dig for it. Homepages are the extreme case, which is [why a homepage is the wrong page to hand an assistant](/homepage-wrong-page-for-ai/).

## Step 6: fix the cheap wins first

Sort by ratio ascending and look at the bottom ten. In practice the recurring causes are inline page builder JSON, a slider or gallery script embedded in the page rather than loaded, comment markup on pages nobody comments on, and related-posts blocks that duplicate content already on the page.

Removing wrapper is usually easier than writing more. A page that goes from 2 percent to 8 percent did not gain content, it lost noise.

## If you would rather not run scripts

The script above is the honest way to learn what your pages look like from the outside. If you want it watched rather than sampled, RankReady is a WordPress AI SEO plugin that audits these same signals from the admin: heading order, content position, Markdown endpoints with correct cache headers, and crawler access.

It runs alongside Rank Math, Yoast, AIOSEO and SEOPress instead of competing with them, since they own titles, meta and sitemaps while this owns machine readability. Setup is about five minutes.

What that gets you is pages that are reachable and cheap to parse. That is a crawlability claim, not a promise about citations, and the difference matters. The plugin lives on [WordPress.org](https://wordpress.org/plugins/rankready-ai-llm-seo/). If the ratios come back fine but your writing is dense, the next measurement is in [measuring how quotable your writing is](/measure-quotable-writing/).

Run it on 25 of your own pages. What is your worst ratio, and is that page one you actually care about?