Which AI crawlers RankReady supports

RankReady manages robots.txt rules for 29 named AI crawlers. That is the number. It is not 31, and it is not “30 plus”. If you see a larger figure anywhere, it is wrong.

This page lists the crawlers documented by name, explains who operates them and what they are generally for, and covers the two mirrored groups that behave differently from the rest.

The named crawlers

Purpose is described in general terms: training means collecting content for model training datasets, search means indexing for a search or answer product, and live retrieval means fetching a page at the moment a user asks something. Operators publish their own documentation and change it from time to time, so treat the purpose column as a guide rather than a contract.

The table above lists the agents documented by name. The full set RankReady manages is 29. Open AI Visibility, Robots subtab in your own install to see the complete list exactly as your version ships it, since that screen is the authoritative view.

The mirrored groups: Googlebot and Facebook’s link crawler

Two entries in the managed block behave unlike the rest, and it is worth understanding why before you assume something changed.

Googlebot and Facebook’s link crawler each get their own mirrored group in robots.txt. A mirrored group repeats the rules your site already has under User-agent: *, written out again under the specific agent name.

The important part: their permissions do not change. The mirrored group says exactly what the wildcard group already said. Nothing is newly allowed and nothing is newly blocked.

So why write it out at all? Because agent-readiness scanners and audit tools look for a named group. When they see only User-agent: *, some report the agent as unaddressed. Naming it makes the existing position explicit and machine-readable. This is a visibility change in your robots.txt, not a policy change on your site.

Googlebot is included because it is your ordinary search crawler, and blocking it by mistake while tightening AI rules would be a costly error. Repeating your existing rules under its name removes any ambiguity about what it is allowed to do.

Content Signals

Alongside per-crawler allow and disallow rules, RankReady writes Content Signals directives into the managed block. There are three:

• ai-train: whether your content may be used to train models.
• search: whether your content may be indexed for search.
• ai-input: whether your content may be used as input to generate an answer.

These express intent separately from access. A crawler can be allowed to fetch a page while you signal that the content should not be used for training. Whether an operator honours a signal is up to that operator.

What these controls actually do

Robots rules and Content Signals are requests, not enforcement. A well-behaved crawler reads them and complies. A badly behaved one ignores them, and only server-level blocking stops it.

Equally, allowing a crawler does not make anything happen. It removes an obstacle. When you later see a hit from one of these agents in Insights, that hit proves your page was fetched. It does not prove the page was quoted anywhere.

All of these settings live in one place: AI Visibility, Robots subtab. RankReady only ever edits the block between # BEGIN RankReady and # END RankReady, so any rules you wrote yourself elsewhere in robots.txt are left untouched.