--- title: "What Cloudflare’s Crawler Default Change Means for a WordPress Blog" url: https://hostmy.blog/cloudflare-crawler-default-wordpress/ date: 2026-09-20 modified: 2026-09-03 lang: en author: "Aditya Sharma" description: "Cloudflare now blocks Training and Agent AI crawlers by default on ad-bearing pages. The decision moved off your server." categories: - "AI Search" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1882-1024x538.jpg word_count: 1554 --- # What Cloudflare’s Crawler Default Change Means for a WordPress Blog On 15 September 2026 Cloudflare changed its defaults: for domains newly onboarding, Training and Agent crawlers are blocked on pages that display ads, while Search stays allowed. The same change evaluates multi-purpose crawlers on all of their behaviours, so blocking Training now also blocks Googlebot, Applebot and BingBot for anyone who chose that. For a WordPress blog behind Cloudflare, that changes where the decision happens. It is no longer made by your robots.txt file being read and obeyed. It is made at the edge, before the request reaches your server, and your access log will not record it. Three consequences follow, and the first one is the reason most people will not notice. ## A refusal at the edge leaves no trace on your origin When Cloudflare declines a request, your server never sees it. There is no line in `/var/log/nginx/access.log`. There is no entry in any WordPress plugin's crawler report, because WordPress never ran. So the usual diagnostic gets you the wrong answer. You grep the log, find nothing, and conclude that no AI crawler has visited. The truthful conclusion is narrower: no AI crawler request reached your origin. `grep -icE "gptbot|claudebot|perplexitybot|oai-searchbot|ccbot" /var/log/nginx/access.log` Zero on that command means one of three things. Nothing visited, the log rotated, or something upstream is answering on your behalf. Rule out the second by checking the rotated files, then investigate the third. A refusal that never reaches WordPress A refusal that never reaches WordPress AI crawler GET /post/ Cloudflare edge bot management WordPress origin access log Accept: text/markdown not forwarded 403 issued at the edge origin log stays empty Your robots.txt says Allow. Nothing inside WordPress can overrule a decision made before the request arrives. The origin access log is the usual evidence for whether a crawler arrived. When the edge answers first, that log stays empty and the absence looks like disinterest. ## Check whether your site is behind Cloudflare at all before anything else Plenty of people inherit a Cloudflare setup they did not configure, through a host or an agency. `curl -sSI https://example.com/ | grep -iE 'server|cf-ray|cf-cache-status'` A `cf-ray` header means Cloudflare is in the path. No `cf-ray` means the rest of this post is background reading rather than an action item. `dig +short example.com` If the addresses returned belong to Cloudflare rather than your host, DNS is proxied and every request goes through the edge first. ## Mixed-use is the category that catches ordinary blogs The ad-page default targets Training and Agent crawlers, and a second change judges multi-purpose crawlers on all of their behaviours at once. Two conditions, and a normal WordPress blog with display ads can meet both without anyone deciding it should. Mixed-use describes a crawler that serves more than one purpose, which covers most of them. Ad-bearing is a property of your page rather than of your intent. The upshot is that a blog running ads may now be declining crawler traffic it never chose to decline, which makes [whether to let AI crawlers in at all](/should-you-block-ai-crawlers/) a decision worth making on purpose. That is not a criticism of the default. It is a statement about where the setting lives, which is no longer in your plugin. ## Content Signals is the new place the preference is expressed Alongside it, Content Signals in robots.txt expresses what a crawler may do with the content rather than only whether it may fetch it. Cloudflare is explicit that these values state a preference rather than issuing blocks directly, so the enforcement still lives in your Cloudflare settings. The shape of the directive looks like this, alongside the ordinary rules: `User-agent: * Content-Signal: search=yes, ai-train=no Allow: /` Verify the exact field names against Cloudflare's own current documentation before you deploy anything. This is a young specification and the field list is the part most likely to move. The important behavioural point is that Content Signals expresses a preference about use, and it sits in a file that also expresses access rules. Those are two separate statements and both need to be right. ## The most-specific-wins rule breaks more robots.txt files than anything else robots.txt matching is most-specific-wins, not additive. A named crawler group replaces the wildcard group entirely. So this file does not do what it looks like it does: `User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Disallow: /?s= Content-Signal: search=yes, ai-train=no User-agent: GPTBot Allow: /blog/` GPTBot matches the second group and only the second group. Every disallow in the first group stops applying to it, and so does the Content Signal line. Your search pages, your cart and your admin paths are all now open to that one agent, and you have said nothing to it about training. The fix is to make each named group complete, or to have no named groups at all. [The full crawler list and the group trap behind it](/block-ai-crawlers-robots-txt/) covers the same rule across every agent worth naming. `User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Disallow: /?s= Content-Signal: search=yes, ai-train=no Allow: / User-agent: GPTBot Disallow: /wp-admin/ Disallow: /cart/ Disallow: /?s= Content-Signal: search=yes, ai-train=no Allow: /blog/` Verbose, and correct. Verbose and correct beats short and wrong in a file that machines read literally. ## Tell an edge block apart from an origin block in three commands Both produce a refused crawler. They are fixed in completely different places, so identifying which one you have is the whole job. [A short reachability check you can run on any post](/ai-crawler-test/) is the quicker version if you only want a yes or no. `curl -sS -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.4' https://example.com/your-post/ curl -sS -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.4' https://example.com/robots.txt curl -sS -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.4' https://example.com/wp-content/uploads/example.jpg` Then read your origin access log for those three requests. | Pattern | Where the block is | | ------- | ------------------ | | Nothing appears in the origin log | The edge answered, so it is a Cloudflare or CDN rule | | Requests appear in the log with 403 or 429 responses | Your own web server or a security plugin | | HTML refused, robots.txt and sitemap fine, images also refused | Web server layer, above WordPress | That third pattern is not hypothetical, and [a host-level refusal nobody was told about](/host-blocking-ai-crawlers/) is more common than the market admits. Measured on our own infrastructure, a site returned 429 to GPTBot while its robots.txt said Allow. The refusal reproduced 3 times out of 3, and static assets were refused too while robots.txt and the sitemap both returned 200, which placed the block at the web server layer rather than inside WordPress. The lesson from that case generalises. The file everybody checks is usually the file that is fine. ## No WordPress plugin can override an edge decision This is worth stating plainly, because the market will tell you otherwise. A plugin runs inside WordPress. WordPress runs on your origin. A request refused at a CDN edge never reaches either. Any plugin advertising that it can guarantee AI crawler access is describing something outside its own execution context. What a plugin can honestly do is control everything from the origin outward: the robots.txt content including Content Signals, whether a lighter Markdown version of the page exists, whether schema is in the raw HTML, and whether freshness signals are accurate. That is the crawlability layer, and it is real work with a real ceiling. More on where that ceiling sits is in [what an assistant's visit looks like from the origin](/ai-crawlers-request-by-request/). ## What is worth doing this week Four steps, in order, and none of them takes long. Check for `cf-ray` and find out whether you are behind Cloudflare. Read your robots.txt and count how many named agent groups exist, then make each one complete. Add or correct the Content Signal line after checking the current field names. Then run the three-command test above against a real post and read your origin log. If the log stays empty, the conversation moves to your Cloudflare dashboard rather than your plugin settings. That is a better place to be than guessing. Payload is the other thing worth fixing while you are in here. Measured on our own infrastructure, one page returned 4.7 MB of HTML against 18 KB of the same content in Markdown, roughly 260 times the payload. Serving a lighter version helps the clients that ask for one, and [who actually reads Markdown copies, and how to serve them safely](/markdown-version-of-blog-posts/) is worth settling first. Markdown negotiation is honoured by coding agents only: Claude Code, Copilot Chat and CLI, Cursor, Microsoft Copilot, OpenClaw and OpenCode. ChatGPT browse, Claude.ai, Perplexity, Gemini and Grok do not ask for it. ## Handling the origin side without editing files by hand The edge stays outside any plugin's reach, RankReady included, worth knowing before you install anything to fix crawler access. What it does cover is everything from the origin outward: robots.txt with Content Signals and complete named groups, a Markdown endpoint, Article and FAQPage schema in the raw HTML, freshness signals, an author box and llms.txt. About five minutes to set up, alongside Rank Math, Yoast, AIOSEO or SEOPress with no conflict, they keep your titles, meta descriptions and sitemap. It is on [the WordPress.org directory](https://wordpress.org/plugins/rankready-ai-llm-seo/), and the four-gate diagnostic is in [the quick curl pass over the same ground](/ai-seo-checker-curl/). ## One thing to find out today Run the `cf-ray` check on your own domain. If Cloudflare is in the path, when did you last look at what it decides on your behalf, and would you have chosen the same default?