--- title: "How to allow GPTBot, ClaudeBot and PerplexityBot" url: https://hostmy.blog/docs/rankready/allow-gptbot-claudebot-perplexitybot/ date: 2026-08-24 modified: 2026-08-25 lang: en author: "Aditya Sharma" description: "Allow specific AI crawlers in your WordPress robots.txt while restricting others, and verify each group is written correctly." word_count: 786 --- # How to allow GPTBot, ClaudeBot and PerplexityBot By the end of this page your `robots.txt` will explicitly allow the AI crawlers you want, restrict the ones you do not, and you will be able to read the file and confirm each group says what you intended. ## Why allow them explicitly If your robots.txt says nothing about a crawler, it falls under whatever your `User-agent: *` group says. That usually works, but it leaves your position implicit. Naming the crawlers you want makes it explicit, survives later edits to the wildcard group, and gives scanners something concrete to read. It also forces a useful decision. GPTBot, ClaudeBot and PerplexityBot are not the same thing as each other, and treating them as one category is how sites end up with a policy nobody chose. ## Know what you are allowing RankReady covers 29 named AI crawlers, so you can be specific. Three of the most commonly asked about: - **GPTBot**, from OpenAI. Note that OpenAI also operates **ChatGPT-User** and **OAI-SearchBot**, which are separate agents. Allowing one does not allow the others, so decide on all three. - **ClaudeBot**, from Anthropic. Related agents include **Claude-Web** and **anthropic-ai**. Same point applies. - **PerplexityBot**, from Perplexity. Others you can set individually include Google-Extended, GoogleOther, Applebot-Extended, Bingbot, Meta-ExternalAgent, FacebookBot, MistralAI-User, Bytespider, Amazonbot, cohere-ai, DuckAssistBot, YouBot, PhindBot, CCBot, AI2Bot, Diffbot and PetalBot. ## How group matching works, which trips people up A crawler reads `robots.txt` and follows the single most specific group that matches its own name. It does not merge that group with the wildcard group. So if you write a named group for GPTBot, that group is the whole of GPTBot's instructions, and your `User-agent: *` rules no longer apply to it. The practical consequence is that a named group must repeat anything you still want to keep out. If your wildcard group disallows a private directory and your GPTBot group does not mention it, GPTBot is now permitted to fetch that directory. Review any named group as a complete policy for that crawler, not as an addition to an existing one. This is also why RankReady gives Googlebot and Facebook's link crawler their own mirrored group that repeats your existing `User-agent: *` rules. Their permissions are unchanged. The group exists so scanners find a named entry, and mirroring the wildcard rules is what keeps behaviour identical. ## Set it in RankReady - In wp-admin, open **RankReady**. - Go to the **AI Visibility** tab, ending in `tab=crawlers`. - Open the **Robots** subtab, `sub=robots`. - Set GPTBot, ClaudeBot and PerplexityBot to allow. - Decide separately on ChatGPT-User, OAI-SearchBot, Claude-Web and anthropic-ai. They are listed individually for a reason. - Set the rest of the list according to your policy. - Save. RankReady writes everything between `# BEGIN RankReady` and `# END RankReady` in your robots.txt. Rules outside those markers are untouched. Do not hand edit inside them, because the block is regenerated from your settings. ![RankReady crawler controls showing individual allow and disallow settings for named AI crawlers](https://hostmy.blog/wp-content/uploads/2026/08/rr-robots-crawlers-v2-scaled.png)Each crawler is set individually in the Robots subtab. ## Verify each group Read the live file rather than trusting the settings screen: `curl -s https://your-site.com/robots.txt` Then check the specific groups. Print the group line and the lines that follow it: `curl -s https://your-site.com/robots.txt | grep -A3 -i 'User-agent: GPTBot' curl -s https://your-site.com/robots.txt | grep -A3 -i 'User-agent: ClaudeBot' curl -s https://your-site.com/robots.txt | grep -A3 -i 'User-agent: PerplexityBot'` For each one, confirm the rules underneath match what you chose, and that anything you still want restricted is repeated inside the group. Do not check with a loose grep for `Disallow: /`. It substring matches ordinary lines such as `Disallow: /wp-admin/` and will make an allowed site look blocked. Match the exact line instead: `curl -s https://your-site.com/robots.txt | grep -cE '^Disallow: /$'` A result of `0` means nothing disallows the whole site. ## Give them something clean to fetch Allowing a crawler only settles permission. What it receives is a normal HTML page with your theme, menus and scripts wrapped around the article. If you also enable the machine readable endpoints in the same tab, the same content is available in a much plainer form: - `/llms.txt`, an index of the site - `/{post-slug}.md`, individual posts as Markdown - `/.well-known/mcp.json`, the [WebMCP manifest](https://hostmy.blog/docs/rankready/how-to-add-webmcp-wordpress/) None of these require an [AI provider](https://hostmy.blog/docs/rankready/choose-ai-provider-cost/) key. ## Set expectations correctly Allowing a crawler means you have permitted a fetch. That is all it means. It does not make anything fetch your site, and if you later see GPTBot or ClaudeBot in your access logs, that hit proves your page was fetched. It does not prove the content was quoted, stored or shown to anyone. Anybody telling you otherwise is selling something. ## Related If your position is more detailed than allow or disallow, [Content Signals](https://hostmy.blog/docs/rankready/content-signals-robots-txt/) let you state what your content may be used for using the `ai-train`, `search` and `ai-input` directives. RankReady writes those into the same managed block.