--- title: "How to block AI crawlers in WordPress" url: https://hostmy.blog/docs/rankready/how-to-block-ai-crawlers-wordpress/ date: 2026-08-24 modified: 2026-08-25 lang: en author: "Aditya Sharma" description: "Use RankReady to write robots.txt rules for 29 named AI crawlers, then verify the managed block is live on your site." word_count: 772 --- # How to block AI crawlers in WordPress By the end of this page your `robots.txt` will carry explicit rules for named AI crawlers, written and maintained by RankReady, and you will know how to read the result rather than trust the interface. ## How blocking actually works `robots.txt` is a request, not a lock. It is a file at the root of your domain that names user agents and tells them which paths they may fetch. Well behaved crawlers read it and comply. A crawler that ignores it faces no technical obstacle, because nothing about the file prevents a request from being made. Be clear about that before you start. If your requirement is enforcement rather than a stated preference, robots.txt is the wrong tool and you need blocking at the server or CDN. If your requirement is to state your position in the standard, machine readable place that operators check, robots.txt is exactly the right tool and RankReady manages it for you. ## What RankReady writes RankReady owns a clearly delimited block inside your `robots.txt`: `# BEGIN RankReady ...rules managed by the plugin... # END RankReady` Everything outside those markers is left alone. Rules written by your SEO plugin, your host or you personally stay where they are. Everything inside is regenerated from your settings, so do not hand edit between the markers. Your changes will be overwritten the next time the block is rebuilt. ## The crawlers it covers RankReady covers **29 named AI crawlers**. The list includes GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Google-Extended, GoogleOther, Applebot-Extended, Bingbot, Meta-ExternalAgent, FacebookBot, MistralAI-User, Bytespider, Amazonbot, cohere-ai, DuckAssistBot, YouBot, PhindBot, CCBot, AI2Bot, Diffbot and PetalBot. They are listed individually rather than lumped together because they do different jobs. Some fetch a page because a user asked a question right now. Some collect content for training. Some are search crawlers from companies that also build models. Blocking all of them with one switch is a decision you should make deliberately, not by accident. ### Googlebot and Facebook's link crawler These two get their own mirrored group in the managed block. The group repeats the rules your site already has under `User-agent: *`. Their permissions do not change in any way. They are named explicitly so that scanners checking for a named group find one, instead of reporting an absence. If you see Googlebot listed in your robots.txt after enabling this, nothing has been restricted. ## Turn it on in RankReady - In wp-admin, open **RankReady**. - Go to the **AI Visibility** tab. The address ends in `tab=crawlers`. - Open the **Robots** subtab, `sub=robots`. - Set each crawler to allow or disallow according to your policy. - Save. Work through the list rather than using a blanket setting. It is a short exercise once, and it is the difference between a policy you can explain and a policy you inherited from a default. ![RankReady AI crawler controls listing named crawlers with allow and disallow settings](https://hostmy.blog/wp-content/uploads/2026/08/rr-robots-crawlers-v2-scaled.png)Per crawler controls in the Robots subtab under AI Visibility. ## Verify it worked Read the live file. Do not rely on the settings screen, because a virtual robots.txt can be overridden by a real file on disk. `curl -s https://your-site.com/robots.txt` Check for these three things: - The `# BEGIN RankReady` and `# END RankReady` markers are present. - The crawlers you chose to disallow appear as their own `User-agent` groups with the rules you expect. - Nothing outside the markers has been altered. To confirm a single crawler quickly, print the group and the few lines after it: `curl -s https://your-site.com/robots.txt | grep -A3 -i 'User-agent: GPTBot'` One warning on checking your own work. Do not grep loosely for `Disallow: /`. That substring matches ordinary WordPress lines such as `Disallow: /wp-admin/` and will convince you the whole site is blocked when it is not. If you want to check for a site wide disallow, match the exact line: `curl -s https://your-site.com/robots.txt | grep -cE '^Disallow: /$'` A result of `0` means no group disallows the entire site. ## If robots.txt does not show your changes - Check whether a physical `robots.txt` file exists in your web root. A real file wins over anything WordPress generates. Rename it and retry. - Clear your page cache and CDN. robots.txt is cached aggressively. - Go to **Settings, Permalinks** and click Save Changes to rebuild rewrite rules. - Check whether another SEO plugin is also writing to robots.txt. RankReady stays inside its own markers, but another plugin may not. ## What comes next Allow and disallow are not the only choices available in this file. [Content Signals](https://hostmy.blog/docs/rankready/content-signals-robots-txt/) let you state what your content may be used for rather than only who may fetch it, using the `ai-train`, `search` and `ai-input` directives. If your policy is more nuanced than a straight block, read the page on Content Signals before finalising your settings.