---
title: "How to Verify a Real AI Crawler Visit: IP Lists and Reverse DNS"
url: https://hostmy.blog/verify-ai-crawler-visits/
date: 2026-11-03
modified: 2026-11-04
lang: en
author: "Aditya Sharma"
description: "A user-agent name proves nothing. How to verify an AI crawler visit with each vendor's published IP list and Google's reverse DNS method, with commands."
categories:
  - "RankReady"
image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2794-1024x538.jpg
word_count: 1616
---

# How to Verify a Real AI Crawler Visit: IP Lists and Reverse DNS

To verify an AI crawler visit, check the address that made the request against the crawler's published IP list, or, for Google, with a reverse and forward DNS lookup. The name in the user-agent string is not enough, because anyone can send any name. A log full of ClaudeBot and OAI-SearchBot lines is a log of things that claim to be those crawlers.

This post shows how to tell the real from the claimed, using what each company publishes. It covers Google, OpenAI, Perplexity and Anthropic, with a command for each method, and says what to do with the result. It does not repeat how to read a log, which [reading your server log by hand](/read-server-log-ai-crawlers/) already covers. It also does not decide whether you should allow a crawler, which [blocking or allowing 29 AI crawlers](/block-ai-crawlers-robots-txt/) does.

![Numbered outline of this post, How to Verify a Real AI Crawler Visit: IP Lists and Reverse DNS: 1 Why a name proves nothing; 2 What each company publishes; 3 Step two: check one address against a list; 4 When your log shows the wrong address; 5 User-triggered fetchers behave differently; 6 What this cannot tell you.](https://hostmy.blog/wp-content/uploads/2026/10/hmb-diagram-2794.png)The route through this post.

## Why a name proves nothing

A user-agent string is text the visitor chooses to send. A scraper can call itself Googlebot, ClaudeBot or anything else, and a log analyser that counts by name will believe it. Google's own documentation says this is the reason verification exists: it helps if you are concerned that spammers or other troublemakers are accessing your site while claiming to be from Google.

The consequences run both ways. If fakes are counted as real, your crawler numbers are inflated and any conclusion about who visits you is wrong. If you allow a name in a firewall rule, a fake with that name walks through. And if you block by name alone, a real crawler with a slightly different string slips past. Checking the address closes all three gaps.

## The two methods

| Method | How it works | Who supports it |
| ------ | ------------ | --------------- |
| A published IP list | You compare the request's address with a list the company publishes | Google, OpenAI, Perplexity and Anthropic |
| Reverse and forward DNS | You look up the name behind the address, then look up that name and confirm it returns the same address | Google documents it |

The list method is the same for every vendor, so it is the one to learn first. DNS is a second method that Google documents for its crawlers, and it is worth knowing because it needs no list.

## What each company publishes

| Crawler family | What is published | Where |
| -------------- | ----------------- | ----- |
| Google crawlers | IP range files, and the DNS method | Google's [guide to verifying Google crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) |
| OpenAI: OAI-SearchBot, GPTBot, ChatGPT-User | One IP list file for each | openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json |
| Perplexity: PerplexityBot, Perplexity-User | One IP list file for each | perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json |
| Anthropic: ClaudeBot and its other agents | One IP list file | claude.com/crawling/bots.json |

Perplexity's documentation describes its files as the source of truth for allowlisting at a firewall. Anthropic's support article says that if a crawler has a source address on its list, that indicates the crawler is coming from Anthropic, and the list is the only verification method it names. Each file we looked at is a JSON document with a creation time and a list of prefixes, written as address ranges. OpenAI's own crawler documentation, covered in our OAI-SearchBot guide, also recommends allowing requests from its published ranges.

## Step one: get the addresses from your log

Start with the addresses that claim to be the crawler you care about. This prints each address that used the name, with a count, most frequent first. The log path varies by host.

`grep -i "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head`

In the common log format the address is the first field. If your site sits behind a CDN or proxy, the first field may be the proxy's address and not the visitor's, which is covered below. Swap the name for ClaudeBot, PerplexityBot or any other. The method of finding and reading the log is in the guide linked at the top of this post.

## Step two: check one address against a list

This downloads a published list and prints whether an address falls inside any of its ranges. The address on the end is a documentation example, so replace it with one from your log.

`curl -s https://openai.com/searchbot.json | python3 -c 'import sys,json,ipaddress; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(sys.stdin); print(any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in d["prefixes"]))' 203.0.113.9`

It prints True if the address is inside a published range and False if not. To test another vendor, change the file address to that vendor's list. We ran this command against OpenAI's published file with one address from inside a range and one from outside, and it returned True and False as expected. A False for an address that used the crawler's name is the result you are looking for: it is a fake, or something sharing the name.

## Google: reverse and forward DNS

Google documents a manual check that needs no list. Run a reverse lookup on the address from your log with the host command. Confirm the name it returns ends in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that name and check it returns the original address.

`host THE_ADDRESS_FROM_YOUR_LOG
host THE_NAME_THAT_CAME_BACK`

Google says common crawlers resolve to names under googlebot.com, special-case crawlers to google.com, and user-triggered fetchers to google.com or gae.googleusercontent.com. The forward step matters. Anyone who controls an address can set its reverse name to anything, so the name only counts if the forward lookup points back to the same address. Google also publishes IP range files for the same purpose, which suits a firewall or a script better than a manual lookup.

## When your log shows the wrong address

A CDN, a reverse proxy or a security service sits between the visitor and your server. If your server logs the proxy's address, every visit looks like it came from the same few places, and none will match a crawler's list. Check whether your logs record the original address, which most setups pass in a header and a log format can capture. If you are not sure, look at the addresses at the top of your log: a short list of addresses owned by one company is a sign you are seeing the proxy. Fix the log format first, because verification means nothing on the wrong address.

## What the result means

| Result | What it means | What to do |
| ------ | ------------- | ---------- |
| On the list, and you want this crawler | A real visit from a crawler you allow | Nothing |
| On the list, and you do not want it | A real crawler, ignoring or outside your wishes | Use robots.txt, and check what it covers |
| Not on the list, using the name | A fake or a look-alike | Block it at the firewall, or ignore it |
| Not on the list, and it ignores robots.txt | A scraper | Block it at the firewall |

The second row has a trap that Anthropic's support article spells out. It warns that blocking by address may not work correctly or persistently guarantee an opt-out, because it stops the crawler from reading your robots.txt in the first place. For a real crawler you want to refuse, the right tool is robots.txt, as [WordPress robots.txt: what to allow, block and test](/wordpress-robots-txt/) explains. Reserve address blocks for the fakes.

## User-triggered fetchers behave differently

Some visits are made because a person asked. Perplexity's documentation says its Perplexity-User fetcher generally ignores robots.txt rules because a user requested the fetch, and OpenAI describes ChatGPT-User in similar terms. Verifying them still works, because they use published lists too, but a robots.txt rule will not stop them. If you want to refuse them, the lever is a firewall rule on the verified address, which is a deliberate choice with its own cost.

## Keep the lists fresh

Published lists change. Each file carries a creation time. When we looked, OpenAI's ChatGPT-User, Anthropic's and Perplexity's PerplexityBot lists had been updated within the past few weeks, while OpenAI's search list showed a much older date (2 January 2026). Do not copy ranges into a rule and forget them. Fetch the file when you run the check, or on a schedule if you automate it, and treat a failed download as a failure to verify, not as a pass. A list that cannot be downloaded proves nothing.

## A routine

`1. Pick one crawler and list the addresses that used its name.
2. Check each address against that vendor's published list.
3. Count how many are real and how many are not.
4. For real visits, read the status codes, as in the five-curl test.
5. For fakes, decide whether to block them at the firewall.
6. Repeat monthly, and after any firewall or CDN change.`

The status-code step is where the two halves meet. A verified crawler receiving a 403 is the case that matters most. This post explains it: [test whether AI crawlers can reach your site, in five curl commands](/ai-crawler-test/). Hosts can also block them without telling you.

## What this cannot tell you

Verification says who made a request, not whether they will honour your rules, and not what they do with the content afterwards. A new crawler that is not on any list yet will fail a list check, so a False means not verified, and not necessarily hostile. And a verified address only proves the visit came from that company's network, which is enough for the decisions above.

## Where RankReady fits

RankReady keeps a log of which AI crawler fetched which URL, and it lets you allow or block 29 crawlers one at a time. A log that records the name a request sent should be read with the point in this post in mind, because a name is a claim. Use the log to see which crawlers are asking, and use the published lists to confirm the ones that matter. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress, and the free [AI Readiness Checker](https://hostmy.blog/plugins/rankready/ai-readiness-checker/) runs 41 checks on your live site.

If you checked the top ten addresses in your log against a published list tonight, how many would be real?