---
title: "Bytespider and CCBot: Block or Allow on WordPress?"
url: https://hostmy.blog/bytespider-ccbot-wordpress/
date: 2026-11-12
modified: 2026-11-12
lang: en
author: "Aditya Sharma"
description: "Bytespider and CCBot differ in who runs them and how verifiable they are. What each says about itself, the disputed evidence, and a block-or-allow guide."
categories:
  - "RankReady"
image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2810-1024x538.jpg
word_count: 1437
---

# Bytespider and CCBot: Block or Allow on WordPress?

Bytespider and CCBot are two crawlers that arrive on a WordPress site with very different reputations. CCBot belongs to Common Crawl, a non-profit that publishes an open archive of the web, and it documents how to verify it. Bytespider belongs to ByteDance, and the evidence about how it behaves is disputed. Whether to block either is a decision you can make from what each says about itself, plus the reports from outside.

This post compares the two. For CCBot it uses Common Crawl's own page. For Bytespider we could not find ByteDance's own documentation in a form we could read, so what it says comes from a news report and a trade summary. The post labels it that way. The log lines shown are invented demonstration lines.

![Numbered outline of this post, Bytespider and CCBot: Block or Allow on WordPress?: 1 CCBot, in Common Crawl](https://hostmy.blog/wp-content/uploads/2026/10/hmb-diagram-2810.png)The route through this post.

## A note on the demo lines

The log lines in this post are invented examples. They use each crawler's name, addresses from the documentation range 203.0.113.0/24 and made-up paths and sizes. They are not from our server and they show no real traffic or behaviour. The CCBot user-agent string is the one Common Crawl publishes. The Bytespider line uses the token only, because the full strings are only reported second-hand.

## CCBot, in Common Crawl's words

Common Crawl's [CCBot page](https://commoncrawl.org/ccbot) describes a non-profit foundation that produces and maintains an open repository of crawl data, with the stated aim of democratising access to web information. The data is meant to be freely available for research and collaboration. The documented user-agent string is CCBot/2.0 with a link to its FAQ. To stop it, the page gives a robots.txt group that names CCBot and disallows the whole site. The page implies CCBot follows robots.txt without saying so outright, and it does not mention Crawl-delay.

The page also covers verification. It says CCBot now runs on dedicated IP ranges with reverse DNS, which lets you check that a logged request is the real thing. Its example is an address resolving to a name under crawl.commoncrawl.org. It points to the published ranges, in both IPv4 and IPv6, as a JSON file at index.commoncrawl.org/ccbot.json. It notes that reverse DNS is not available over IPv6. And it warns that some crawlers falsely claim to be CCBot, recommending that you verify user-agent strings.

## Bytespider, as reported

ByteDance, the company behind TikTok, operates Bytespider. A trade summary at [PPC Land](https://ppc.land/bytespider/) says it crawls public pages and sends the content back for Toutiao Search indexing, for recommendation systems and for training large language models. It describes the crawler as dominated by training use. The same summary says ByteDance's webmaster documentation states that Bytespider follows the Robots Exclusion Protocol and tells operators how to block it with a two-line rule. We could not read that documentation directly, so treat this as a report of what the documentation says.

The evidence on compliance is where the two crawlers part. The summary reports that Fortune, in October 2024, reported that Bytespider ignores robots.txt. It reports TollBit research on the first half of 2026, published in August 2026. It found that Bytespider was one of three agents that reached disallowed pages on close to half of the European sites that named it in robots.txt. It also reports administrators saying the crawler stops on disallowed paths once it re-fetches an updated file. It suggests the gap between publishing a rule and the crawler complying explains some of the apparent violations. The summary does not date or source that last point, so we do not rely on it.

On verification, the summary says ByteDance publishes no signed verification method like Google's, so anyone can send the Bytespider name, and counts by user agent alone may misattribute traffic. It lists address blocks that ByteDance publishes, and notes that observed traffic often comes from outside them.

## Side by side

| Question | CCBot | Bytespider |
| -------- | ----- | ---------- |
| Who runs it | Common Crawl, a non-profit | ByteDance |
| What for | An open archive of the web | Search indexing, recommendations and model training, as reported |
| Documented user agent | CCBot/2.0 with a link to its FAQ | Variants reported, not confirmed by us |
| Says it follows robots.txt | Implied on its page | Reported to say so |
| Evidence of compliance | Not examined here | Disputed, per the reports above |
| How to verify | Published IP ranges and reverse DNS | No signed method, per the summary |

The honest summary is that CCBot is a crawler with a documented identity and a documented way to check it. Bytespider is a crawler whose operator's own documentation we could not read and whose behaviour is contested. That affects what blocking can achieve. A robots.txt rule works on a crawler that obeys it.

## What invented lines look like

`203.0.113.40 - - [22/Oct/2026:11:20:03 +0000] "GET /robots.txt HTTP/1.1" 200 612 "-" "CCBot/2.0 (https://commoncrawl.org/faq/)"
203.0.113.40 - - [22/Oct/2026:11:20:05 +0000] "GET /sample-post/ HTTP/1.1" 200 18452 "-" "CCBot/2.0 (https://commoncrawl.org/faq/)"
203.0.113.41 - - [22/Oct/2026:11:22:44 +0000] "GET /private-area/ HTTP/1.1" 200 4100 "-" "Mozilla/5.0 (compatible; Bytespider)"`

The third line is the one to read twice. In this invented example the crawler asked for a path that the robots.txt in the first line disallows, and was served. In real life that pattern would suggest the rule was not honoured, or that the request came from something using the name. You would need to check the address before drawing either conclusion, and the test in [do AI crawlers respect robots.txt? A log-based test](/ai-crawlers-respect-robots-txt/) shows how.

## Count them in your own log

`grep -ci "CCBot" /var/log/nginx/access.log
grep -ci "Bytespider" /var/log/nginx/access.log
grep -i "Bytespider" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head`

The first two count lines per crawler. The third lists the addresses behind the Bytespider name, which is where the verification starts. For CCBot, [how to verify a real AI crawler visit: IP lists and reverse DNS](/verify-ai-crawler-visits/) shows how to check an address against a published list. The log path varies, as [reading your server log by hand](/read-server-log-ai-crawlers/) explains.

## Block or allow

| If you | Then |
| ------ | ---- |
| Do not want your content in an open web archive | Block CCBot in robots.txt |
| Are happy to contribute to an open archive | Allow CCBot |
| Do not want your content used by ByteDance's systems | Block Bytespider in robots.txt, and add a firewall rule if the log shows it ignoring the file |
| Have no view | Allow CCBot, and watch Bytespider's load |

One piece of context for the CCBot rows. PPC Land reports that the News/Media Alliance sent Common Crawl a formal demand for content removal on 29 April 2026. We have not read the demand and do not know how Common Crawl responded, so treat it as a reason to decide on purpose, not as a verdict.

The second part of the Bytespider answer is practical. A robots.txt rule is the first step, and if the log shows requests to disallowed paths after the file has been fetched, the next step is a firewall rule for the verified addresses. That is a different case from [Anthropic's ClaudeBot](/claudebot-wordpress/), where address blocking is discouraged because it stops the crawler reading the file. For a crawler that does not honour the file, there is nothing to read. The WordPress steps are in [blocking or allowing 29 AI crawlers, and the robots.txt trap that exposes you](/block-ai-crawlers-robots-txt/), and the file itself is in [WordPress robots.txt: what to allow, block and test](/wordpress-robots-txt/).

## What this post cannot tell you

It cannot tell you whether either crawler visits your site, how often, or whether Bytespider would honour your file. The demo lines are invented, the CCBot page describes intent, and the Bytespider evidence is second-hand and disputed. Your log answers the question for your site. We have not published our own findings, and we make no claim about them.

## A routine

- Count each crawler's lines in your log.

- Verify the addresses behind each name.

- For the verified ones, compare the paths asked for with your robots.txt.

- Decide on each crawler separately.

- If a block is ignored, move to a firewall rule on the verified addresses.

## Where Host My Blog fits

Host My Blog documents RankReady, a free WordPress plugin that lets you allow or block 29 AI crawlers one at a time. It writes the robots.txt rules with your global rules repeated inside them, and keeps a log of which AI crawler fetched which URL. A log that records the name a request sent should be read with the verification point in mind. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress, and the free [AI Readiness Checker](https://hostmy.blog/plugins/rankready/ai-readiness-checker/) runs 41 checks on your live site.

If Bytespider asked for a path your robots.txt disallows, would your log tell you?

## Sources

- Common Crawl, CCBot page. [https://commoncrawl.org/ccbot](https://commoncrawl.org/ccbot) (read 2026-11-11).

- PPC Land, Bytespider summary, citing ByteDance webmaster documentation, Fortune (3 October 2024) and TollBit research (reported 14 August 2026). Secondary source. [https://ppc.land/bytespider/](https://ppc.land/bytespider/) (read 2026-11-11).

- Common Crawl, published IP ranges for CCBot. [https://index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json) (checked 2026-11-11).