--- title: "AI crawler control for WordPress" url: https://hostmy.blog/plugins/rankready/ai-crawler-control-wordpress/ date: 2026-09-03 modified: 2026-09-03 lang: en author: "Rajendra Zore" description: "Decide which AI crawlers may read your blog, with Content Signals written into robots.txt. Runs alongside Rank Math, Yoast and AIOSEO." word_count: 861 --- # AI crawler control for WordPress Twenty nine named AI crawlers, each with its own toggle, written into your robots.txt alongside Content Signals. About five minutes from the WordPress dashboard, no developer, and nothing your SEO plugin writes is touched. v1.3.1 · 300 active installs · Tested to WordPress 7.1 · Runs alongside Yoast, Rank Math, AIOSEO and SEOPress ## Doing nothingis still a decision. Every AI crawler reaching your site right now is operating under whatever your robots.txt happens to say, which on most WordPress installs is a wildcard rule written for Googlebot years ago. Two things changed underneath that. ![The AI Crawlers grid in RankReady, with Allow, Default and Block radio columns and a bulk setter above, listing named crawlers grouped by vendor, starting with OpenAI's GPTBot annotated as GPT model training data and ChatGPT-User.](https://hostmy.blog/wp-content/uploads/2026/09/rr130-crawler-grid.png) ### Crawlers stopped wanting the same thing GPTBot trains a model. OAI-SearchBot builds an answer someone is reading now. ChatGPT-User is a person who clicked. Treating those three identically made sense when they did not exist. ### The control surface moved Cloudflare began blocking mixed use AI crawlers by default on ad bearing pages, including for existing free customers, and moved permissions to Content Signals. If you have never expressed an intent, something is expressing one for you. ### Most specific match, not additive The moment a crawler gets its own group it stops inheriting your wildcard rules. Name Bingbot to give it one instruction and you have silently dropped your admin path rules for it. ## A switch per crawler,written in safely. Grouped by vendor, with allow, default and block on every row, and a set all control at the top for when you want one position across the board. ![The Enable Crawler Rules control in RankReady, a checkbox that appends LLM crawler rules to robots.txt, described as adding per-crawler user-agent blocks and auto-syncing to the physical robots.txt file for 29 named AI crawlers.](https://hostmy.blog/wp-content/uploads/2026/09/rr130-crawler-enable.png) #### 29 crawlers, individually Answer engines, training crawlers and search indexers, each one a switch. Allow the ones you want quoting you, block the ones you do not, and nothing is decided for you by default. #### Content Signals, expressed separately search, ai-input and ai-train as three answers rather than one, so you can say index me and quote me but do not train on me. #### Global rules repeated into every named group This is the part almost nobody gets right. RankReady copies your wildcard rules into each named crawler group, so naming a bot cannot silently detach it from your admin and billing path rules. #### Fenced inside its own block Everything between the BEGIN and END markers is managed. Everything outside it is yours and is never touched. Deactivate the plugin and the block goes while yours remains. #### Works with a virtual robots.txt WordPress serves one and RankReady writes into it. A real file on disk takes precedence, which is a common reason changes appear to do nothing, and the plugin says so. #### Googlebot gets its own mirrored group Rather than joining the permissive AI group, where most specific match would hand it your admin-ajax and search pages. ## robots.txt is a request,not a wall. Two limits worth knowing before you trust any of this, including one we found on our own site. #### Well behaved crawlers obey it, others do not If your goal is enforcement rather than expression, the tool is a block at the server or the CDN, not a text file. #### Your host may already be overriding you On one of our own sites robots.txt said Allow: / for GPTBot while the server returned 429 to it, because a hosting firewall rule nobody knew about. The file said one thing and the server did another. #### Blocking a training bot does not remove you from answers GPTBot, OAI-SearchBot and ChatGPT-User are separate switches, so declining training and staying readable in answers is a configuration, not a compromise. #### A crawler hit is not a citation The log tells you the page was fetched. It does not tell you that you were quoted, and the plugin says exactly that rather than implying otherwise. ## The honest answers,before you have to ask. ## Make your site readableto the things reading it. RankReady is a WordPress AI SEO plugin. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress rather than replacing any of them, and setup takes about five minutes from your dashboard. Free, no account, no credit card · GPL-2.0 · About five minutes from your dashboard ## Questions people actually ask ### If I block GPTBot, do I disappear from ChatGPT? No. GPTBot is the training crawler. OAI-SearchBot builds answers for ChatGPT search and ChatGPT-User fetches a page when a person clicks. Those are separate switches here. ### Does blocking training crawlers hurt my SEO? Not your Google rankings. Google-Extended controls AI training use and is documented as separate from Search indexing. Googlebot is a different switch, which is exactly why they are not the same toggle. ### Will this conflict with the robots.txt my SEO plugin writes? No. RankReady writes inside a fenced block and leaves everything else alone. Yoast, Rank Math, AIOSEO and SEOPress all keep their own output. ### Do AI crawlers actually respect robots.txt? The major ones publish that they do and behave accordingly in our logs. Some do not. Treat the file as a clear statement of intent and use server level blocking if you need enforcement. ### How do I tell whether any of this is working? Watch the crawler log rather than the settings screen. RankReady records which AI bots fetched which post and when. ### How long does setup take? About five minutes. Open AI Visibility, set the crawlers you care about, save.