--- title: "Headless WordPress and SEO: What AI Crawlers Can Read" url: https://hostmy.blog/headless-wordpress-seo/ date: 2026-10-23 modified: 2026-10-23 lang: en author: "Aditya Sharma" description: "A headless WordPress front end builds the HTML a crawler reads. Here is what AI crawlers can see, how to check it with curl, and what to fix first." categories: - "RankReady" image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2570-1024x538.jpg word_count: 1280 --- # Headless WordPress and SEO: What AI Crawlers Can Read A headless WordPress site keeps WordPress as the place where you write. A separate front end, built with a framework such as Next.js or Astro, sits in front of it. That can be a good architecture. It also moves a job that WordPress used to do for you, producing the HTML a crawler reads, to code you now own. That job matters more than it used to. Google renders JavaScript, but the AI crawlers that matter for assistants may not, and a headless site that builds its pages in the browser can look empty to them. This post is about that one question, what a crawler can read from a headless WordPress site, and how to check it. It is not about the framework choices or the build. ## What changes when WordPress goes headless In a normal WordPress site, the server returns finished HTML. Your SEO plugin prints the title, the canonical and the schema into it. In a headless setup, WordPress returns data, usually through its REST API or GraphQL, and the front end turns that data into a page. Everything an SEO plugin used to place in the page now has to be placed by the front end. | Job | Normal WordPress | Headless | | --- | ---------------- | -------- | | Building the HTML | WordPress and the theme | The front end framework | | Title, canonical, meta description | The SEO plugin prints them | The front end has to read them from the API and print them | | Structured data | A plugin prints it | The front end has to output it | | robots.txt and sitemap | WordPress serves them | Whatever serves the public domain does, which may no longer be WordPress | | Redirects | A plugin or a server rule | The front end or its host | The plugin that used to do each of these may still hold the data. Nothing guarantees that it reaches the page. ## What AI crawlers read The finding that matters comes from a study by Vercel and MERJ, published in December 2024. It reported that none of the major AI crawlers it measured rendered JavaScript. GPTBot, ClaudeBot and PerplexityBot fetched JavaScript files, but did not run them. The practical consequence in Vercel's write-up is to server-render the content that matters, using server-side rendering, incremental static regeneration or static generation. It says to leave client-side rendering for features that do not carry content. You can read it in [the rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler). Two cautions. The study is almost two years old, and crawler behaviour changes, so treat it as a reason to test, not a final answer. And Googlebot is different: Google renders JavaScript, so a client-rendered page can rank while remaining unreadable to the crawlers feeding other assistants. That gap is easy to miss because the site looks fine in a browser and in Google. ## Check what a crawler receives Browsers show you the page after the scripts have run. A crawler that does not run scripts sees only what the server sent. Fetch the page with no rendering, and look for a sentence from the body of the post. `curl -s https://yourdomain.com/your-post/ | grep -c 'a sentence from the body of your post'` A result of 1 or more means the text is in the HTML. A result of 0 means it appears only after JavaScript runs, and a crawler that does not run scripts will not see it. Do the same for the title and the schema. `curl -s https://yourdomain.com/your-post/ | grep -o '[^<]*' curl -s https://yourdomain.com/your-post/ | grep -c 'application/ld+json'` The method is the same as in [finding every page an AI cannot parse](/find-pages-ai-cannot-parse/) and [the five curl commands](/ai-crawler-test/). Run it on a handful of different templates, not only the homepage, because a single route that renders differently is enough to hide a section of the site. ## What to fix, in order | Problem | What to look for | Fix | | ------- | ---------------- | --- | | Body text missing from the HTML | The grep above returns 0 | Render that route on the server or at build time | | No title or canonical in the HTML | They appear only after scripts run | Output them from the front end's server-side layer | | Schema missing | No JSON-LD in the raw HTML | Inject it on the server from the data the API returns | | robots.txt points at the wrong site | The public domain serves no robots.txt, or serves WordPress's | Serve the right file from the public domain, and see the robots.txt guide | | Sitemap lists WordPress URLs | It lists the backend domain | Generate it for the public domain | A headless site also has a second public face to protect. If the WordPress back end is reachable on its own domain, make sure it is not indexed as a copy of the site. The file side of that is in [our WordPress robots.txt guide](/wordpress-robots-txt/), and the structure of the data a page should carry is in [structured data in WordPress](/structured-data-wordpress/). ## The ways a headless front end can render | Mode | What a crawler receives | What to watch | | ---- | ----------------------- | ------------- | | Static generation | Finished HTML built ahead of time | Pages are stale until the next build | | Server-side rendering | Finished HTML built on each request | Server load, and slow responses if the API is slow | | Incremental regeneration | Finished HTML that is refreshed on a schedule or on demand | A page can be out of date until it refreshes | | Client-side rendering | A near-empty shell, with the content added by scripts | The case that can look empty to a crawler that does not run scripts | The first three put the content in the HTML. The last does not. Vercel's advice, quoted above, is to use the first three for anything that carries content, and to keep client-side rendering for features that do not. ## The back end is a second site A headless setup has two public faces: the front end that readers see, and the WordPress back end on its own address. If the back end is reachable, it can be crawled and indexed as a copy of the site. Check that it is blocked from indexing, that preview and draft routes are not public, and that links in the content point at the front end's addresses and not the back end's. A post that links to the wrong domain sends readers, and crawlers, to a place you did not mean them to go. ## A test to run before launch `1. Fetch five different templates with curl and no scripts. 2. Search each for a sentence from the body. 3. Check each has one title, one canonical and one JSON-LD block. 4. Fetch robots.txt and the sitemap from the public domain. 5. Fetch the back end's address and confirm it is not indexable.` Do this before launch, and again after any front-end release. A framework upgrade, or a change to how one route is rendered, can quietly move content from the HTML into the browser. ## Should you go headless for SEO? No. Going headless does not improve search results by itself. It also adds work that a normal WordPress site does for free, because WordPress returns finished HTML and your SEO plugin writes the tags. There are good reasons to go headless, such as sharing content with an app or a team that works in a framework. Better crawling is not one of them. If you do it, the checks above are the price of keeping what you already had. ## Where RankReady fits RankReady's job on a headless site is limited. In a normal setup it writes AI-facing output into the page. In a headless setup the front end has to carry it, so RankReady exposes its data, such as summaries, FAQs and ready-to-use schema, to your front end through a read-only API. That is a Pro feature, and its details are on [the headless API page](/plugins/rankready/headless-api-wordpress/), not repeated here. The free [AI Readiness Checker](https://hostmy.blog/plugins/rankready/ai-readiness-checker/) reads your public site the way a crawler does, which makes it a quick way to see what comes through. If you fetched your most important post right now without running any JavaScript, would you still find your own first paragraph in it?