---
title: "ClaudeBot: What It Crawls and Whether to Block It"
url: https://hostmy.blog/claudebot-wordpress/
date: 2026-11-09
modified: 2026-11-09
lang: en
author: "Aditya Sharma"
description: "ClaudeBot is Anthropic's training crawler, and it is one of three Anthropic bots. What each does, what blocking costs, and how to spot it in a log, with demo lines."
categories:
  - "RankReady"
image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2808-1024x538.jpg
word_count: 1379
---

# ClaudeBot: What It Crawls and Whether to Block It

ClaudeBot is Anthropic's crawler for collecting web content that could help train its AI models, and whether to block it is a decision about training, not about visibility. Anthropic runs two other bots, Claude-User and Claude-SearchBot, with different jobs, and treating the three as one is how site owners end up blocking more than they meant to.

This post sets out what each Anthropic bot does, what happens to your site if you block each one, and how to recognise ClaudeBot in your own log. It is built on Anthropic's own support article. The log lines you will see are invented demonstration lines, explained below, and not records from our server.

![Numbered outline of this post, ClaudeBot: What It Crawls and Whether to Block It: 1 The three Anthropic bots; 2 What Anthropic says about how the bots behave; 3 What ClaudeBot looks like in a log; 4 Deciding whether to block ClaudeBot; 5 Do not block by address; 6 Verify before you act.](https://hostmy.blog/wp-content/uploads/2026/10/hmb-diagram-2808.png)The route through this post.

## A note on the demo lines

The log lines in this post are invented examples. They use the crawler's name, addresses from the documentation range 203.0.113.0/24 and made-up paths and sizes. They are not from our server and they show no real traffic, no real volume and no real behaviour. Anthropic's article does not publish a full user-agent string for any of its bots, so the lines use only the bot's name as a token. Their purpose is to show you what to look for in your own log.

## The three Anthropic bots

Anthropic's [article on its crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) names three, with a purpose and a consequence for each.

| Bot | What it is for | If you block it |
| --- | -------------- | --------------- |
| ClaudeBot | Collecting web content that could help train Anthropic's generative AI models | Anthropic says this signals that your site's future material should be excluded from training datasets. |
| Claude-User | Visiting a site when a Claude user asks a question that needs it | Claude cannot retrieve your content for user queries, which may reduce visibility in user-directed search |
| Claude-SearchBot | Navigating the web to improve search result quality, and checking content for relevance and accuracy | Your content is not indexed for search, which may reduce visibility and accuracy in user search results |

The first is about training. The other two are about being found and used when someone asks. Blocking the first is a stance on how your writing is used. Blocking the others has a visibility cost, which is why they deserve a separate decision.

## What Anthropic says about how the bots behave

The article lists principles. Data collection should be transparent. Crawling should not be intrusive or disruptive, and Anthropic says it limits how quickly it crawls the same domains and respects Crawl-delay where appropriate. The bots respect do-not-crawl signals by honouring industry-standard robots.txt directives, and they respect anti-circumvention technologies, with Anthropic giving the example that it will not try to bypass CAPTCHAs. What the article does not say is how the bots treat password-protected or login-gated pages, beyond that last principle, so do not assume anything about them.

That is a statement of intent and policy from Anthropic. It is not a measurement. Whether any crawler behaves as its operator says is something you can test on your own site, as [do AI crawlers respect robots.txt? A log-based test](/ai-crawlers-respect-robots-txt/) sets out.

## What ClaudeBot looks like in a log

Here are three invented lines in the common combined log format. The first asks for robots.txt, which is what a well-behaved crawler does before anything else. The second fetches a post and is served. The third is refused by something on the server.

`203.0.113.20 - - [22/Oct/2026:09:14:07 +0000] "GET /robots.txt HTTP/1.1" 200 612 "-" "Mozilla/5.0 (compatible; ClaudeBot)"
203.0.113.20 - - [22/Oct/2026:09:14:09 +0000] "GET /sample-post/ HTTP/1.1" 200 18452 "-" "Mozilla/5.0 (compatible; ClaudeBot)"
203.0.113.21 - - [22/Oct/2026:09:15:31 +0000] "GET /sample-post/ HTTP/1.1" 403 199 "-" "Mozilla/5.0 (compatible; ClaudeBot)"`

Read them left to right. The first field is the address, then the time, then the request, the status code, the size, the referrer and the user agent. The status code is the part to watch. A 200 means the page was served. A 403 on a crawler you did not mean to refuse is the kind of thing [your host may be blocking AI crawlers without telling you](/host-blocking-ai-crawlers/) describes. Because the name is only a token, it proves nothing about who sent the request, which is why the last section covers verification.

## Count it in your own log

Three commands answer the first questions about any crawler: how often it visited, what it asked for and how it was answered. Replace the log path with yours.

`grep -ci "ClaudeBot" /var/log/nginx/access.log
grep -i "ClaudeBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head
grep -i "ClaudeBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn`

The first counts the lines that carry the name. The second lists the paths it asked for, most frequent first. The third counts the status codes. The field numbers assume the common log format, and [reading an nginx or Apache access log for AI crawler traffic, properly](/access-log-ai-crawler-analytics/) explains how to check yours.

## Deciding whether to block ClaudeBot

Blocking ClaudeBot is a defensible position if you do not want your writing used in model training, and the cost is small because it does not affect the two bots that matter for visibility. Allowing it is a defensible position if you are comfortable with it. There is no neutral default. The article says that to block it entirely you add a rule to the robots.txt in your top-level directory, naming the bot with a disallow for the whole site. It adds that you must do it for every subdomain you want to opt out. The step-by-step for WordPress is in [blocking or allowing 29 AI crawlers, and the robots.txt trap that exposes you](/block-ai-crawlers-robots-txt/), and the general file is in [WordPress robots.txt: what to allow, block and test](/wordpress-robots-txt/). Those are not repeated here.

If your concern is load and not training, the article describes a lighter option: Anthropic supports the non-standard Crawl-delay directive to limit crawl rate, with a one-second delay as its example. It is worth noting that Crawl-delay is not part of the robots standard, so it works only where a crawler chooses to honour it, and Anthropic says it does.

## Do not block by address

The article gives one warning that is easy to miss. Blocking by IP address may not work correctly or persistently guarantee an opt-out, because it prevents the bots from reading your robots.txt in the first place. A crawler that cannot fetch the file cannot learn that you opted out. Use the robots.txt rule for an opt-out, and use address rules, if at all, for traffic you have confirmed is not Anthropic's.

## Verify before you act

Anthropic publishes a list of its bots' IP addresses at claude.com/crawling/bots.json, and says that if a crawler's source address is on that list, it is from Anthropic. A request using the ClaudeBot name from an address that is not on the list is not verified. [How to verify a real AI crawler visit: IP lists and reverse DNS](/verify-ai-crawler-visits/) gives the command. Do that before you decide anything from a count, because the count is only as good as the identification.

## What the demo lines cannot tell you

They cannot tell you how often ClaudeBot visits your site, what it fetches or whether it respects your file, because they are invented. Anthropic's article cannot tell you either, since it describes intent. Those answers come from your own log, and the three commands above are how you get them. We have not published our own ClaudeBot findings here, and a post claiming them would need real log access.

## A routine

`1. Run the count command and note the number.
2. Check the top addresses against Anthropic's published list.
3. Read the status codes for the verified ones.
4. Decide on ClaudeBot, Claude-User and Claude-SearchBot separately.
5. Re-check after any security or host change.`

## Where RankReady fits

RankReady lets you allow or block 29 AI crawlers one at a time, including Anthropic's. It writes the robots.txt rules with your global rules repeated inside them, and keeps a log of which AI crawler fetched which URL. Setting the three Anthropic bots separately is the point of a per-crawler control. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress, and the free [AI Readiness Checker](https://hostmy.blog/plugins/rankready/ai-readiness-checker/) runs 41 checks on your live site.

If you counted ClaudeBot in your own log tonight, would you know which of the three Anthropic bots you were looking at?