---
title: "Which AI crawlers RankReady supports"
url: https://hostmy.blog/docs/rankready/supported-ai-crawlers/
date: 2026-08-24
modified: 2026-08-25
lang: en
author: "Aditya Sharma"
description: "The AI crawlers RankReady names in robots.txt, who runs them, and how the mirrored Googlebot and Facebook groups work."
word_count: 671
---

# Which AI crawlers RankReady supports

RankReady manages robots.txt rules for **29 named AI crawlers**. That is the number. It is not 31, and it is not "30 plus". If you see a larger figure anywhere, it is wrong.

This page lists the crawlers documented by name, explains who operates them and what they are generally for, and covers the two mirrored groups that behave differently from the rest.

![The RankReady robots and crawlers screen listing AI crawler user agents with permission controls](https://hostmy.blog/wp-content/uploads/2026/08/rr-robots-crawlers-v2-scaled.png)Every crawler below is toggled from AI Visibility, Robots subtab.

## The named crawlers

Purpose is described in general terms: *training* means collecting content for model training datasets, *search* means indexing for a search or answer product, and *live retrieval* means fetching a page at the moment a user asks something. Operators publish their own documentation and change it from time to time, so treat the purpose column as a guide rather than a contract.

| User agent | Operator | General purpose |
| ---------- | -------- | --------------- |
| `GPTBot` | OpenAI | Training |
| `ChatGPT-User` | OpenAI | Live retrieval |
| `OAI-SearchBot` | OpenAI | Search indexing |
| `ClaudeBot` | Anthropic | Training |
| `Claude-Web` | Anthropic | Live retrieval |
| `anthropic-ai` | Anthropic | Training |
| `PerplexityBot` | Perplexity | Search indexing |
| `Google-Extended` | Google | A robots token, not a separate fetcher. It controls whether Google may use your content for AI training and grounding. |
| `GoogleOther` | Google | Non-search crawling for internal and research uses |
| `Applebot-Extended` | Apple | A robots token controlling whether Apple may use your content for AI training |
| `Bingbot` | Microsoft | Search indexing |
| `Meta-ExternalAgent` | Meta | Training and indexing |
| `FacebookBot` | Meta | Training |
| `MistralAI-User` | Mistral AI | Live retrieval |
| `Bytespider` | ByteDance | Training |
| `Amazonbot` | Amazon | Search indexing and product services |
| `cohere-ai` | Cohere | Training and retrieval |
| `DuckAssistBot` | DuckDuckGo | Live retrieval for DuckAssist answers |
| `YouBot` | You.com | Search indexing |
| `PhindBot` | Phind | Search indexing |
| `CCBot` | Common Crawl | Open dataset collection, widely reused for training |
| `AI2Bot` | Allen Institute for AI | Research dataset collection |
| `Diffbot` | Diffbot | Structured data extraction |
| `PetalBot` | Huawei, Petal Search | Search indexing |

The table above lists the agents documented by name. The full set RankReady manages is 29. Open AI Visibility, Robots subtab in your own install to see the complete list exactly as your version ships it, since that screen is the authoritative view.

## The mirrored groups: Googlebot and Facebook's link crawler

Two entries in the managed block behave unlike the rest, and it is worth understanding why before you assume something changed.

Googlebot and Facebook's link crawler each get their **own mirrored group** in robots.txt. A mirrored group repeats the rules your site already has under `User-agent: *`, written out again under the specific agent name.

The important part: **their permissions do not change**. The mirrored group says exactly what the wildcard group already said. Nothing is newly allowed and nothing is newly blocked.

So why write it out at all? Because agent-readiness scanners and audit tools look for a named group. When they see only `User-agent: *`, some report the agent as unaddressed. Naming it makes the existing position explicit and machine-readable. This is a visibility change in your robots.txt, not a policy change on your site.

Googlebot is included because it is your ordinary search crawler, and blocking it by mistake while tightening AI rules would be a costly error. Repeating your existing rules under its name removes any ambiguity about what it is allowed to do.

## Content Signals

Alongside per-crawler allow and disallow rules, RankReady writes [Content Signals](https://hostmy.blog/docs/rankready/content-signals-robots-txt/) directives into the managed block. There are three:

- `ai-train`: whether your content may be used to train models.
- `search`: whether your content may be indexed for search.
- `ai-input`: whether your content may be used as input to generate an answer.

These express intent separately from access. A crawler can be allowed to fetch a page while you signal that the content should not be used for training. Whether an operator honours a signal is up to that operator.

## What these controls actually do

Robots rules and Content Signals are requests, not enforcement. A well-behaved crawler reads them and complies. A badly behaved one ignores them, and only server-level blocking stops it.

Equally, allowing a crawler does not make anything happen. It removes an obstacle. When you later see a hit from one of these agents in Insights, that hit proves your page was fetched. It does not prove the page was quoted anywhere.

All of these settings live in one place: AI Visibility, Robots subtab. RankReady only ever edits the block between `# BEGIN RankReady` and `# END RankReady`, so any rules you wrote yourself elsewhere in robots.txt are left untouched.