---
title: "Read an nginx or Apache Access Log for AI Crawler Traffic, Properly"
url: https://hostmy.blog/access-log-ai-crawler-analytics/
date: 2026-09-23
modified: 2026-09-23
lang: en
author: "Aditya Sharma"
description: "Your access log already knows which AI crawlers visit and what they get refused. Here is how to read it without misleading yourself."
categories:
  - "AI Search"
image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1890-1024x538.jpg
word_count: 1691
---

# Read an nginx or Apache Access Log for AI Crawler Traffic, Properly

Analytics will never show you AI crawler traffic. Google Analytics runs JavaScript, and crawlers do not execute it. The only honest record of who fetched what is your access log.

That log is already on your server. It answers questions no dashboard can: which crawlers actually visit, how often, what they get served, and what they get refused. It is the only record of [each hop between an AI answer and your origin](/ai-crawlers-request-by-request/) that you own outright.

It also misleads you badly if you run the obvious commands. This post covers both.

## Where the log is

`ls -la /var/log/nginx/access.log
ls -la /var/log/apache2/access.log
ls -la /var/log/httpd/access_log`

One of those exists depending on your stack. On managed hosting the path is often per site, something like `/home/yoursite/logs/access.log`. Rotated files sit alongside as `.1`, `.2.gz` and so on.

To search across rotations including the compressed ones:

`zgrep -h 'GPTBot' /var/log/nginx/access.log* | wc -l`

`zgrep` reads plain and gzipped files, `-h` suppresses the filename prefix so the line still parses as a log line.

## The field positions you need

Combined log format, which is the default on both nginx and Apache, looks like this:

`34.25.94.150 - - [25/Aug/2026:00:04:36 +0000] "GET /post/ HTTP/1.1" 200 11994 "-" "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)"`

Split on whitespace and the useful fields are `$1` client IP, `$4` timestamp, `$7` path, `$9` status code, `$10` bytes sent, and everything from `$12` onward the user agent.

Confirm your own format before trusting those numbers. Some hosts prepend a hostname or a request ID, which shifts every field:

`head -1 /var/log/nginx/access.log`

If the line starts with something other than an IP address, add the offset to every field number below. If field arithmetic is more than you want, [a shorter set of greps for the same file](/read-server-log-ai-crawlers/) gets you the headline without the awk.

## The command everyone runs, and why it lies

Here is the natural first attempt:

`awk '$0 ~ /GPTBot|ClaudeBot|PerplexityBot/ {print $1, $9, $7}' access.log \
| sort | uniq -c | sort -rn | head -5`

Run against a sample log, it produces this. Every figure in this post comes from the practice generator at the end, with a fixed random seed, so you can reproduce all of them exactly:

` 1 66.97.164.251 200 /blog/schema-guide/
1 66.94.76.20 301 /blog/ai-crawler-test/
1 66.93.175.102 404 /blog/old-post/
1 66.92.56.75 301 /blog/old-post/
1 66.91.72.216 200 /robots.txt`

Every count is 1. That is not a finding, that is a broken aggregation.

The cause is `$1`. Crawlers arrive from large IP ranges and almost never repeat an address, so including the IP makes every row unique and `uniq -c` has nothing to collapse. The command runs, exits clean, and tells you nothing.

Drop the IP and the same data becomes readable:

`awk '/GPTBot|ClaudeBot|PerplexityBot/ {print $9, $7}' access.log \
| sort | uniq -c | sort -rn | head -10`

```
128 200 /wp-json/wp/v2/posts
121 200 /robots.txt
116 200 /blog/ai-crawler-test/
113 200 /
95 200 /blog/schema-guide/
83 200 /llms.txt
70 200 /blog/pricing/
65 403 /blog/pricing/
51 403 /llms.txt
51 301 /blog/old-post/
```

Same log, same filter, one field removed, and now it is a report. Keep the IP only when you are verifying a specific crawler's identity, which is a different job.

## Counting each crawler separately

Matching the whole line is quick but sloppy: a referrer or a path containing the word GPTBot would match too. Read the user agent field explicitly.

`awk '{
ua = "";
for (i = 12; i <= NF; i++) ua = ua $i " ";
split("GPTBot ChatGPT-User OAI-SearchBot ClaudeBot Claude-User PerplexityBot \
CCBot Bytespider Amazonbot Google-Extended Applebot-Extended meta-externalagent", b, " ");
for (k in b) if (index(ua, b[k])) n[b[k]]++
}
END { for (x in n) printf "%-22s %d\n", x, n[x] }' access.log | sort -k2 -rn`

Expected output:

`GPTBot 470
ClaudeBot 348
PerplexityBot 265
CCBot 213`

That is BSD awk on macOS and GNU awk on Linux, no gawk extensions used.

Two names in that list are there as a reminder rather than a real expectation. Google-Extended and Applebot-Extended are robots.txt control tokens for opting out of AI training, and they are not how the corresponding fetchers identify themselves. A zero against those two is the normal result, so do not read it as evidence that your opt-out is working or failing. Check that in robots.txt instead, using the method in [checking your own file group by group](/audit-robots-txt-by-hand/).

## Status codes per crawler, the number that matters most

A crawler visiting a lot is not the same as a crawler being served.

`awk '/GPTBot/ {print $9}' access.log | sort | uniq -c | sort -rn`

```
280 200
116 403
41 301
31 404
2 500
```

That is 116 refusals out of 470 requests. A crawler being refused a quarter of the time is a configuration finding, and it never shows up anywhere except here.

To see exactly what is being refused:

`awk '/GPTBot|ClaudeBot|PerplexityBot|CCBot/ && ($9 == 403 || $9 == 404 || $9 >= 500) \
{print $9, $7}' access.log | sort | uniq -c | sort -rn | head -5`

```
65 403 /blog/pricing/
58 404 /blog/old-post/
51 403 /llms.txt
12 404 /llms.txt
9 404 /wp-json/wp/v2/posts
```

A 403 on `/llms.txt` is the kind of thing you would otherwise never discover. The file exists, you published it, and [an upstream rule your host never mentioned](/host-blocking-ai-crawlers/) is refusing it to the exact clients it was written for.

To reproduce a 403 from your desk rather than guessing, see [sending the requests yourself before reading the log](/ai-crawler-test/).

## Volume over time, and what it costs you

Requests per day per crawler shows whether something changed:

`awk '/GPTBot|ClaudeBot|PerplexityBot|CCBot/ {
split($4, d, ":"); day = substr(d[1], 2);
for (i = 12; i <= NF; i++)
if ($i ~ /GPTBot|ClaudeBot|PerplexityBot|CCBot/) {
gsub(/[^A-Za-z-]/, "", $i); bot = $i; break
}
print day, bot
}' access.log | sort | uniq -c`

```
42 25/Aug/2026 CCBot
57 25/Aug/2026 ClaudeBot
95 25/Aug/2026 GPTBot
56 25/Aug/2026 PerplexityBot
31 26/Aug/2026 CCBot
54 26/Aug/2026 ClaudeBot
96 26/Aug/2026 GPTBot
```

And the bandwidth, which is the argument you will need if someone asks why you are allowing this traffic:

`awk '/GPTBot|ClaudeBot|PerplexityBot|CCBot/ {b += $10; n++}
END {printf "%d requests, %.1f MB, avg %d bytes\n", n, b/1048576, b/n}' access.log`

```
1296 requests, 47.4 MB, avg 38322 bytes
```

That average byte figure is the same measurement as text-to-HTML ratio, seen from the server side. Cutting page weight cuts it directly, and the method is in [finding the templates that bury their own content](/find-pages-ai-cannot-parse/).

## Which pages they actually take

`awk '/GPTBot|ClaudeBot|PerplexityBot|CCBot/ && $9 == 200 {print $7}' access.log \
| sort | uniq -c | sort -rn | head -8`

```
149 /robots.txt
145 /blog/ai-crawler-test/
144 /wp-json/wp/v2/posts
139 /
115 /blog/schema-guide/
105 /llms.txt
```

Two things to look for. Heavy repeat fetching of `/robots.txt` is normal and healthy. Heavy fetching of your homepage relative to your articles is less useful than it looks, since homepages were cited about 4 percent of the time in a study of 153,425 citations across six platforms. Depth pages are where the value sits.

## Practising safely

You do not need production traffic to learn these commands. Generate a synthetic log and run everything above against it first:

`python3 - > access.log <<'PY'
#!/usr/bin/env python3
"""Generate a synthetic combined-format access log for practice."""
import random, sys
from datetime import datetime, timedelta
random.seed(7)
UAS = [
("GPTBot", 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'),
("ClaudeBot", 'Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)'),
("PerplexityBot", 'Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)'),
("CCBot", 'CCBot/2.0 (https://commoncrawl.org/faq/)'),
("Googlebot", 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'),
("human", 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/128.0 Safari/537.36'),
]
PATHS = ["/", "/blog/ai-crawler-test/", "/blog/schema-guide/", "/llms.txt",
"/robots.txt", "/blog/old-post/", "/wp-json/wp/v2/posts", "/blog/pricing/"]
t = datetime(2026, 8, 25, 0, 3, 11)
rows = []
for _ in range(4000):
t += timedelta(seconds=random.randint(3, 240))
name, ua = random.choices(UAS, weights=[9, 7, 5, 4, 12, 40])[0]
path = random.choice(PATHS)
if name == "GPTBot" and path in ("/llms.txt", "/blog/pricing/"):
code = 403
elif path == "/blog/old-post/":
code = random.choice([200, 301, 404])
else:
code = random.choices([200, 301, 404, 500], weights=[88, 6, 5, 1])[0]
size = 0 if code in (301, 404) else random.randint(2500, 90000)
ip = "%d.%d.%d.%d" % (random.choice([20, 40, 52, 66, 34]),
random.randint(1, 254), random.randint(1, 254),
random.randint(1, 254))
rows.append('%s - - [%s +0000] "GET %s HTTP/1.1" %d %d "-" "%s"'
% (ip, t.strftime("%d/%b/%Y:%H:%M:%S"), path, code, size, ua))
sys.stdout.write("\n".join(rows) + "\n")
PY`

Run the aggregations against that, confirm the numbers move the way you expect, then point them at the real thing.

## Two traps worth naming

User agents are self reported. Anything can claim to be GPTBot, and knowing [which OpenAI agent is which](/gptbot-wordpress/) matters before you act on a count. If a specific crawler's identity matters for a decision, verify the IP against the published ranges each operator maintains, rather than trusting the string.

And your log format may not include the user agent at all. Some hosts trim it. If every crawler command returns zero on a site with real traffic, check `head -1` on the log before concluding you are being ignored. A zero from a log that never recorded the field looks identical to a zero from no visits.

## If you would rather not run scripts

Reading the log by hand is the only way to see what actually happened, and it is worth doing at least once so the numbers stop being abstract.

For ongoing visibility, RankReady is a WordPress AI SEO plugin that records AI crawler hits from inside WordPress, so you get per crawler counts and status codes without SSH access, which matters on hosting where you cannot reach the log at all.

It runs alongside Rank Math, Yoast, AIOSEO and SEOPress rather than replacing them, because it covers a different layer: crawler access, robots.txt groups, Markdown endpoints and schema, while they keep handling titles, meta and sitemaps. Setup takes about five minutes.

The claim stays where it belongs. This tells you whether crawlers reached you and what they received, which is crawlability. It does not tell you whether anyone was cited, and no log and no plugin can. It is on [WordPress.org](https://wordpress.org/plugins/rankready-ai-llm-seo/).

Run the status code breakdown on your own log tonight. What percentage of AI crawler requests came back as something other than 200, and did you know that was happening?