---
title: "Do AI Crawlers Respect robots.txt? A Log-Based Test on WordPress"
url: https://hostmy.blog/ai-crawlers-respect-robots-txt/
date: 2026-11-07
modified: 2026-11-07
lang: en
author: "Aditya Sharma"
description: "How to test whether an AI crawler respects robots.txt in your own log: vendor claims, test design, three outcomes with demo lines, and what not to test."
categories:
  - "RankReady"
image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2815-1024x538.jpg
word_count: 1517
---

# Do AI Crawlers Respect robots.txt? A Log-Based Test on WordPress

An AI crawler respects robots.txt if, after fetching a current copy of your file, it stops requesting the paths you disallowed. You can test that in your own log with one changed rule, a wait of at least a day, and a check of three things. They are the crawler's identity, whether it read the new file, and whether it still asked for the path. The result is evidence about one crawler on one site, which is what a claim needs.

This post is the method. It sets out what each vendor says about robots.txt, how to design the test, how to read the result, and why some crawlers should not be judged by it at all. It uses the vendors' own documentation. We have not run this test here, so every log line is an invented demonstration line. The lines use crawler names as tokens, addresses from the documentation range 203.0.113.0/24 and made-up paths and sizes. None of them shows real traffic, volume or behaviour, and none of the verdicts describes a real crawler.

## What the vendors say

The claims below are each vendor's own account. They are statements of intent, and testing is how you find out whether they hold for you.

| Crawler | What the vendor says about robots.txt |
| ------- | ------------------------------------- |
| Google's common crawlers | They always respect robots.txt rules for automatic crawls, per [Google's documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers) |
| ClaudeBot | Honours industry-standard robots.txt directives, per Anthropic's support article |
| Diffbot | Its crawls adhere to a site's robots.txt by default, and the rule can be overridden in specific cases such as a partnership or agreement, per [Diffbot's FAQ](https://www.diffbot.com/docs/crawl/faq/robots-txt) |
| PerplexityBot | Perplexity documents robots.txt tags for managing it and says changes can take up to 24 hours to take effect, per [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| Perplexity-User | Generally ignores robots.txt, because a user requested the fetch, per [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| Meta-ExternalFetcher | May bypass robots.txt, per [Meta](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) |
| FacebookExternalHit | Might bypass robots.txt for security or integrity checks, per Meta |

The pattern is worth noticing before you test anything. Automatic crawlers generally say they respect the file. Fetchers that act on a person's request generally say they may not. So the test applies to the first kind and not the second. Later posts in this series look at individual crawlers.

## The design

The cleanest test uses a path the crawler has already requested, so you do not have to hope it finds a new one. Add a disallow for that path, for that crawler only, and see whether the requests stop once the crawler has read the new file.

`1. Pick a verified crawler and a path it has requested before.
2. Note the time. Add a disallow for that path, for that crawler only.
3. Confirm the served robots.txt shows the rule.
4. Wait at least 24 hours, because vendors say they cache the file.
5. Search the log for requests after that time.
6. Record whether it fetched robots.txt, and whether it asked for the path.`

The wait in step four comes from the vendors' own documentation. Meta says its crawlers may cache robots.txt for up to 24 hours, and Perplexity says changes can take up to 24 hours. Testing sooner proves nothing, because a crawler that has not yet re-read the file is still following the old one.

## Get the evidence

These commands pull the three pieces out of the log, and [reading your server log by hand](/read-server-log-ai-crawlers/) explains where to find it. Replace the crawler name, the path and the log location.

`grep -i "ClaudeBot" /var/log/nginx/access.log | grep "GET /robots.txt"
grep -i "ClaudeBot" /var/log/nginx/access.log | grep "GET /test-path/"
grep -i "ClaudeBot" /var/log/nginx/access.log | awk '{print $1}' | sort -u`

The first shows when the crawler read your file. The second shows when it asked for the path. The third lists the addresses behind the name, which you then check against the vendor's published list, as in [how to verify a real AI crawler visit: IP lists and reverse DNS](/verify-ai-crawler-visits/). Without that last step, a request by name might be a different visitor wearing the name.

## Three invented outcomes

The lines below show the three results you can get. They are made up, and the dates are arbitrary.

`# A: read the new file, then stayed away
203.0.113.90 [23/Oct 06:02] GET /robots.txt 200
(no later requests for /test-path/)

# B: read the new file, then asked anyway
203.0.113.91 [23/Oct 06:02] GET /robots.txt 200
203.0.113.91 [23/Oct 06:03] GET /test-path/ 200

# C: never read the new file, kept asking
203.0.113.92 [23/Oct 09:40] GET /test-path/ 200`

| Outcome | What it suggests | What to do |
| ------- | ---------------- | ---------- |
| A: read the file, then stopped asking | The crawler honoured the rule | Nothing further |
| B: read the file, then asked for the path | It did not honour the rule, if the address is verified | Check the file is correct, then consider a firewall rule on verified addresses |
| C: asked without reading the file | Inconclusive, because it may be using a cached copy or may not be checking | Wait longer, and repeat |

Outcome B is the only one that points at the crawler, and it points at it only if three conditions hold. The address is verified, the file you served was correct, and enough time has passed. Check the file carefully, because a rule in the wrong group is the commonest reason a rule seems not to work, as [audit your own robots.txt for the most-specific-match trap, by hand](/audit-robots-txt-by-hand/) explains.

Read the second command with care. A request that returns a refusal, such as a 403 from your firewall, is not the same as a request that returns a page. If a rule in your firewall is already blocking the crawler, it never reaches your robots.txt logic and the test says nothing about robots.txt. Check the status code on each line, and note it beside the outcome. A crawler that asks for the path and receives a 200 has fetched the content. One that asks and is refused has not.

It also helps to keep a small record for each run: the crawler name, the verified address, the path, the time you changed the rule, the time you checked, and the outcome. Two or three runs over a few weeks tell you more than one, because a single day can be an outlier. If a crawler honours the rule on Monday and not on Friday, that is worth knowing, and it is only visible if you wrote Monday down.

## Do not test the wrong crawlers

A fetch made because a person asked is not a crawl, and the vendors who run them say as much. Perplexity-User generally ignores robots.txt. Meta-ExternalFetcher may bypass it. If you run this test on those and see requests, the result is what the documentation said it would be, and it is not evidence against the vendor. Apply the test to automatic crawlers only. For a page that must not be fetched by anything, use access control, not a robots rule. [Should you block AI crawlers or let them in](/should-you-block-ai-crawlers/) sets out the wider choice, and [OAI-SearchBot: the crawler behind ChatGPT search](/oai-searchbot-wordpress/) shows one crawler in detail.

## What can go wrong with the test itself

| Problem | Why it ruins the result | How to avoid it |
| ------- | ----------------------- | --------------- |
| The rule is in the wrong group | The crawler matches a different group | Read the served file, and check specificity |
| A real file overrides the virtual one | The rule you wrote is not the one served | Fetch the file as a crawler would |
| A cache serves an old file | The crawler sees the old rule | Clear caches, and check the served file |
| The address is not verified | You may be watching an impostor | Check against the published list |
| You changed too many things | You cannot tell what mattered | Change one rule at a time |
| You tested too soon | The crawler had not re-read the file | Wait at least a day |

## What a result is worth

A clean result for one crawler on one site is a fact about that crawler on that site, at that time. It is not a verdict on the vendor. Crawlers change, and so do their documents. Repeat the test now and then for any crawler whose behaviour matters to you. If one fails the test, the ordinary tool is a firewall rule on verified addresses. If it is a crawler with a contact address in its documentation, you can also message the operator at the address given in its documentation, if it lists one.

## A routine

`1. Choose an automatic crawler you can verify.
2. Choose one path it has requested.
3. Change one rule, and read the served file.
4. Wait a day, and then search the log.
5. Record the outcome, with the date and the address check.
6. Repeat after any change to your file or your CDN.`

## Where RankReady fits

RankReady writes your robots.txt rules, lets you allow or block 29 AI crawlers one at a time, and keeps a log of which AI crawler fetched which URL. The log is a useful place to find the path to test, though a log that records the name a request sent still needs the address check above. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress, and the free [AI Readiness Checker](/plugins/rankready/ai-readiness-checker/) runs 41 checks on your live site.

Sources: [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers), [Meta](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), [Anthropic](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Diffbot](https://www.diffbot.com/docs/crawl/faq/robots-txt) crawler documentation.

If you changed one rule tonight and read your log tomorrow, which of the three outcomes would you expect to see?