Black Friday Sale is Live: Get up to 30% OFF RankReady Pro - the WordPress AI SEO pluginBlack Friday Sale is Live: Up to 30% OFF Get Deal
RankReady

How to Verify a Real AI Crawler Visit: IP Lists and Reverse DNS

3 Nov 2026 8 min read

To verify an AI crawler visit, check the address that made the request against the crawler’s published IP list, or, for Google, with a reverse and forward DNS lookup. The name in the user-agent string is not enough, because anyone can send any name. A log full of ClaudeBot and OAI-SearchBot lines is a log of things that claim to be those crawlers.

This post shows how to tell the real from the claimed, using what each company publishes. It covers Google, OpenAI, Perplexity and Anthropic, with a command for each method, and says what to do with the result. It does not repeat how to read a log, which reading your server log by hand already covers. It also does not decide whether you should allow a crawler, which blocking or allowing 29 AI crawlers does.

Numbered outline of this post, How to Verify a Real AI Crawler Visit: IP Lists and Reverse DNS: 1 Why a name proves nothing; 2 What each company publishes; 3 Step two: check one address against a list; 4 When your log shows the wrong address; 5 User-triggered fetchers behave differently; 6 What this cannot tell you.
The route through this post.

Why a name proves nothing

A user-agent string is text the visitor chooses to send. A scraper can call itself Googlebot, ClaudeBot or anything else, and a log analyser that counts by name will believe it. Google’s own documentation says this is the reason verification exists: it helps if you are concerned that spammers or other troublemakers are accessing your site while claiming to be from Google.

The consequences run both ways. If fakes are counted as real, your crawler numbers are inflated and any conclusion about who visits you is wrong. If you allow a name in a firewall rule, a fake with that name walks through. And if you block by name alone, a real crawler with a slightly different string slips past. Checking the address closes all three gaps.

The two methods

MethodHow it worksWho supports it
A published IP listYou compare the request’s address with a list the company publishesGoogle, OpenAI, Perplexity and Anthropic
Reverse and forward DNSYou look up the name behind the address, then look up that name and confirm it returns the same addressGoogle documents it

The list method is the same for every vendor, so it is the one to learn first. DNS is a second method that Google documents for its crawlers, and it is worth knowing because it needs no list.

What each company publishes

Crawler familyWhat is publishedWhere
Google crawlersIP range files, and the DNS methodGoogle’s guide to verifying Google crawlers
OpenAI: OAI-SearchBot, GPTBot, ChatGPT-UserOne IP list file for eachopenai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json
Perplexity: PerplexityBot, Perplexity-UserOne IP list file for eachperplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json
Anthropic: ClaudeBot and its other agentsOne IP list fileclaude.com/crawling/bots.json

Perplexity’s documentation describes its files as the source of truth for allowlisting at a firewall. Anthropic’s support article says that if a crawler has a source address on its list, that indicates the crawler is coming from Anthropic, and the list is the only verification method it names. Each file we looked at is a JSON document with a creation time and a list of prefixes, written as address ranges. OpenAI’s own crawler documentation, covered in our OAI-SearchBot guide, also recommends allowing requests from its published ranges.

Step one: get the addresses from your log

Start with the addresses that claim to be the crawler you care about. This prints each address that used the name, with a count, most frequent first. The log path varies by host.

grep -i "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

In the common log format the address is the first field. If your site sits behind a CDN or proxy, the first field may be the proxy’s address and not the visitor’s, which is covered below. Swap the name for ClaudeBot, PerplexityBot or any other. The method of finding and reading the log is in the guide linked at the top of this post.

Step two: check one address against a list

This downloads a published list and prints whether an address falls inside any of its ranges. The address on the end is a documentation example, so replace it with one from your log.

curl -s https://openai.com/searchbot.json | python3 -c 'import sys,json,ipaddress; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(sys.stdin); print(any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in d["prefixes"]))' 203.0.113.9

It prints True if the address is inside a published range and False if not. To test another vendor, change the file address to that vendor’s list. We ran this command against OpenAI’s published file with one address from inside a range and one from outside, and it returned True and False as expected. A False for an address that used the crawler’s name is the result you are looking for: it is a fake, or something sharing the name.

Google: reverse and forward DNS

Google documents a manual check that needs no list. Run a reverse lookup on the address from your log with the host command. Confirm the name it returns ends in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that name and check it returns the original address.

host THE_ADDRESS_FROM_YOUR_LOG
host THE_NAME_THAT_CAME_BACK

Google says common crawlers resolve to names under googlebot.com, special-case crawlers to google.com, and user-triggered fetchers to google.com or gae.googleusercontent.com. The forward step matters. Anyone who controls an address can set its reverse name to anything, so the name only counts if the forward lookup points back to the same address. Google also publishes IP range files for the same purpose, which suits a firewall or a script better than a manual lookup.

When your log shows the wrong address

A CDN, a reverse proxy or a security service sits between the visitor and your server. If your server logs the proxy’s address, every visit looks like it came from the same few places, and none will match a crawler’s list. Check whether your logs record the original address, which most setups pass in a header and a log format can capture. If you are not sure, look at the addresses at the top of your log: a short list of addresses owned by one company is a sign you are seeing the proxy. Fix the log format first, because verification means nothing on the wrong address.

What the result means

ResultWhat it meansWhat to do
On the list, and you want this crawlerA real visit from a crawler you allowNothing
On the list, and you do not want itA real crawler, ignoring or outside your wishesUse robots.txt, and check what it covers
Not on the list, using the nameA fake or a look-alikeBlock it at the firewall, or ignore it
Not on the list, and it ignores robots.txtA scraperBlock it at the firewall

The second row has a trap that Anthropic’s support article spells out. It warns that blocking by address may not work correctly or persistently guarantee an opt-out, because it stops the crawler from reading your robots.txt in the first place. For a real crawler you want to refuse, the right tool is robots.txt, as WordPress robots.txt: what to allow, block and test explains. Reserve address blocks for the fakes.

User-triggered fetchers behave differently

Some visits are made because a person asked. Perplexity’s documentation says its Perplexity-User fetcher generally ignores robots.txt rules because a user requested the fetch, and OpenAI describes ChatGPT-User in similar terms. Verifying them still works, because they use published lists too, but a robots.txt rule will not stop them. If you want to refuse them, the lever is a firewall rule on the verified address, which is a deliberate choice with its own cost.

Keep the lists fresh

Published lists change. Each file carries a creation time. When we looked, OpenAI’s ChatGPT-User, Anthropic’s and Perplexity’s PerplexityBot lists had been updated within the past few weeks, while OpenAI’s search list showed a much older date (2 January 2026). Do not copy ranges into a rule and forget them. Fetch the file when you run the check, or on a schedule if you automate it, and treat a failed download as a failure to verify, not as a pass. A list that cannot be downloaded proves nothing.

A routine

1. Pick one crawler and list the addresses that used its name.
2. Check each address against that vendor's published list.
3. Count how many are real and how many are not.
4. For real visits, read the status codes, as in the five-curl test.
5. For fakes, decide whether to block them at the firewall.
6. Repeat monthly, and after any firewall or CDN change.

The status-code step is where the two halves meet. A verified crawler receiving a 403 is the case that matters most. This post explains it: test whether AI crawlers can reach your site, in five curl commands. Hosts can also block them without telling you.

What this cannot tell you

Verification says who made a request, not whether they will honour your rules, and not what they do with the content afterwards. A new crawler that is not on any list yet will fail a list check, so a False means not verified, and not necessarily hostile. And a verified address only proves the visit came from that company’s network, which is enough for the decisions above.

Where RankReady fits

RankReady keeps a log of which AI crawler fetched which URL, and it lets you allow or block 29 crawlers one at a time. A log that records the name a request sent should be read with the point in this post in mind, because a name is a claim. Use the log to see which crawlers are asking, and use the published lists to confirm the ones that matter. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress, and the free AI Readiness Checker runs 41 checks on your live site.

If you checked the top ten addresses in your log against a published list tonight, how many would be real?

Share this
Aditya Sharma
Written by

Aditya Sharma

Host My Blog

Aditya runs Host My Blog and writes The Cited. Five years in WordPress, most of it spent on what actually makes a blog get found rather than what is supposed to. His attention now goes to how AI answers are changing that, and what a blog has to do to still be the source people land on. Everything he writes about here gets tested on the blogs Host My Blog looks after before it reaches the page.

Comments 0

Your email stays private. Be kind.
We’ll handle the technical half

Want your posts in
AI answers, without the work?

Schema, speed and structure are included on every Host My Blog plan. You write; we make sure the machines can read it.

Get tips by email