---
title: "WordPress robots.txt: What to Allow, Block and Test"
url: https://hostmy.blog/wordpress-robots-txt/
date: 2026-10-15
modified: 2026-10-15
lang: en
author: "Aditya Sharma"
description: "Where WordPress gets its robots.txt, what is worth blocking, and how to test the file before a wrong line costs you traffic."
categories:
  - "WordPress"
image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2487-1024x538.jpg
word_count: 1779
---

# WordPress robots.txt: What to Allow, Block and Test

A WordPress site has a robots.txt file whether you wrote one or not, and most owners have never read theirs.

That matters because the file is short, public and trusted completely. One wrong line can ask every search engine to stay away from the whole site, and nothing in your dashboard will tell you it happened.

Here is the answer in one paragraph. Allow what a crawler needs to see your pages the way a visitor does. Block the few paths that generate endless near-identical URLs. Never use the file to hide anything. Then test it, because the file you think you are serving and the file crawlers actually receive are often not the same.

## Where a WordPress robots.txt actually comes from

There are two possibilities, and which one you have decides where you edit it.

If a real file named `robots.txt` sits in the web root, the web server hands it out directly and WordPress never runs. If there is no file, WordPress builds a virtual one on request. Core writes a short default, and the robots.txt editors in Yoast, Rank Math, AIOSEO and SEOPress either extend that virtual file or write a physical one, depending on the plugin and your setup.

The core default is only a few lines. It blocks the admin area, allows the one admin file that front-end features depend on, and points to the sitemap.

`User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml`

That is a sensible file, and many sites would be better off leaving it alone. The trouble starts when a host, a CDN, a migration or a well-meant plugin changes the answer.

Which robots.txt a crawler actually receives

Which robots.txt a crawler actually receives

Crawler asks
for the file
GET /robots.txt

Is there a real file
in the web root?
server checks first

Yes: server sends
that file as is
WordPress never runs

No: WordPress
builds its own
core + plugin rules

yes
no
A CDN, cache or host rule can still replace either response.

Editing the wrong layer is the most common reason a robots.txt change appears to do nothing. Find out which of the two you have before you change anything.

## The two-minute check: is yours already broken?

Before reading more advice, look at what your site serves right now. Request the file and read the status line and the first few rules.

`curl -si https://example.com/robots.txt | head -20`

You want a `200` status, a content type of `text/plain`, and rules you recognise. A `404` is harmless, because crawlers treat a missing file as permission to crawl everything. A server error is not harmless: when robots.txt returns a `5xx`, Google stops crawling the site for the first 12 hours, then falls back to its last good copy of the file while it keeps retrying.

Then look for the two lines that do the most damage.

`User-agent: *
Disallow: /`

That pair tells every compliant crawler to stay out of the entire site. On WordPress it usually comes from one tick box, Settings, Reading, Discourage search engines from indexing this site. It also adds a noindex tag to every page. It is correct on a staging copy and a disaster on the live site, and because it is a database setting it travels with the database when someone copies one site over the other. That is why it sits on [the migration checklist](/how-to-migrate-wordpress-site/), and why a staging copy needs the opposite treatment, covered in [staging done properly](/wordpress-staging/).

## What to allow

Allow whatever a crawler needs to see the page the way a visitor does. Google renders pages, which means it fetches your stylesheets, scripts and images. Old advice to block `/wp-content/` or `/wp-includes/` comes from a time when crawlers read raw HTML only. It is still copied into new files every week, and it can make a page look broken or empty to the system deciding how to rank it.

Keep three things reachable: theme and plugin assets under `/wp-content/`, your uploaded images, and `admin-ajax.php`, which many front-end features call. Then add one `Sitemap` line so crawlers find your sitemap without guessing.

## What is actually worth blocking

Fewer paths than most guides suggest. A path earns a `Disallow` when it creates a very large number of near-identical URLs, or when no search result should ever land there.

| Path or pattern | Block it? | Why |
| --------------- | --------- | --- |
| `/wp-admin/` (keep `admin-ajax.php` allowed) | Yes | No search result belongs in the admin area |
| `/?s=` and `/search/` | Yes | Internal search can generate unlimited URLs from a single page |
| `/cart/`, `/checkout/`, `/my-account/` | Yes, on WooCommerce sites | Personal and transactional pages with nothing to rank |
| `/*?replytocom=` | Yes, if threaded comments are on | Every comment reply creates a duplicate URL of the post |
| `/wp-content/` and `/wp-includes/` | No | Crawlers need the styles and scripts to render the page |
| `/wp-content/uploads/` | No | Your images are how you appear in image search |
| Tag, category and author archives | Not here | Thin archives are a noindex decision, not a robots.txt one |
| `/wp-login.php` | No need | Blocking it hides nothing from an attacker; harden it instead, as in [security without a plugin that fights your host](/wordpress-security/) |

Wildcards are worth knowing. Google supports `*` for any run of characters and `$` for the end of a URL, so `Disallow: /*?replytocom=` blocks reply links on every post with one line. Not every crawler understands them, and a pattern that is slightly wrong blocks more than you meant.

The global group of our own file is short: it blocks the admin area, the client portal and one other private path, allows admin-ajax.php, and points to one sitemap. The file as a whole is longer, because it also names AI crawlers separately, and each of those groups repeats the same rules for the reason explained in the AI crawler section below.

## robots.txt is not a way to hide a page

`Disallow` stops a compliant crawler from fetching a URL. It does not stop the URL being indexed. If other pages link to it, Google can still list the address with no description, which is the opposite of hidden. The file is also public, so a line naming a secret folder advertises the folder.

It causes a quieter mistake too. To keep a page out of results you need a noindex tag, and a crawler can only see that tag if it is allowed to fetch the page. Disallow a page and add noindex, and the noindex is never read. If the goal is removal, allow the crawl and use noindex, as [Google documents it](https://developers.google.com/search/docs/crawling-indexing/block-indexing). If the goal is privacy, put the page behind a login, because robots.txt is a request and not a lock.

## Testing it before it costs you traffic

Testing has three parts, from fastest to most authoritative. The first is the request above: status, content type, content.

The second is a rule check. This short script asks whether a given crawler may fetch a list of paths, using the parser in Python's standard library.

`import sys
from urllib.robotparser import RobotFileParser

site, agent = sys.argv[1], sys.argv[2]
paths = sys.argv[3:]

rp = RobotFileParser(site.rstrip("/") + "/robots.txt")
rp.read()

for path in paths:
verdict = "allowed" if rp.can_fetch(agent, site.rstrip("/") + path) else "BLOCKED"
print(f"{verdict:8} {path}")`

Run against this site as Googlebot, it looks like this.

`python3 robots_check.py https://hostmy.blog Googlebot /blog/ /wp-admin/ /wp-admin/admin-ajax.php "/?s=test"

allowed /blog/
BLOCKED /wp-admin/
allowed /wp-admin/admin-ajax.php
allowed /?s=test`

Read the second and third lines together. The admin area is blocked and the one file inside it that matters is still allowed, because the more specific `Allow` beats the broader `Disallow`. That is the behaviour to confirm on your own site. The last line is also real: we do not block internal search, which a site with more search traffic than ours might want to.

One caution. Parsers differ between languages and versions, so this is a check, not a ruling. The reference for Googlebot is the robots.txt report in Search Console, under Settings, which shows the copy Google last fetched, when it fetched it and any parsing problems. Google's old standalone tester has been retired, so guides that point to it are out of date. Google also caches the file, often for up to a day, so a fix does not take effect the moment you save it.

The third part is repetition. Run all of this again after anything that touches the site's plumbing: a migration, a staging copy going live, a new SEO or caching plugin, a CDN change. Those are the moments the file changes without anyone editing it.

## Layers that quietly change the answer

The file in a plugin screen is not always the file crawlers receive. A physical file can shadow the virtual one, so edits in your SEO plugin appear to do nothing. A caching layer can keep serving an old copy for hours, and ours says so openly: our own response carries a one hour cache header, so a change can take that long to reach everyone.

A CDN or host can add its own rules too. Cloudflare in particular can manage the file for you, which changes what crawlers see without touching WordPress. We covered what that default means in [what Cloudflare's crawler default change means for a WordPress blog](/cloudflare-crawler-default-wordpress/). The rule of thumb is simple: always test the public URL, never the settings screen.

## The AI crawler part, in one paragraph

AI crawlers read the same file, and they follow the same matching rule: each crawler picks the single group that names it best and ignores the rest, including your global rules. A block added for one named crawler can therefore switch off every `Disallow` above it, which we walk through in [auditing your robots.txt for the most specific match trap](/audit-robots-txt-by-hand/). Deciding which of them to let in is a separate question, argued both ways in [blocking or allowing 29 AI crawlers](/block-ai-crawlers-robots-txt/), and you can confirm what actually reaches your server with [five curl commands](/ai-crawler-test/).

## Where Host My Blog fits

Everything above is doable by hand, and most of it takes about twenty minutes once. Read the file, remove what you do not understand, run the check, and set a reminder to repeat it.

What is hard is remembering to repeat it. The file changes silently after a migration, a restore, a plugin swap or a CDN change, and nobody is assigned to look. On the blogs we look after, robots.txt is on the checklist after a restore, as we described in [the backup that did not restore](/wordpress-backup/), because a site that quietly blocks crawlers looks perfectly healthy on every uptime monitor.

If you would rather someone else owned that, it is part of how we run WordPress care [for bloggers](/for-bloggers/) and [for agencies](/for-agencies/). If you would rather keep doing it yourself, you now have the whole method.

Open `yourdomain.com/robots.txt` in a private window right now. Is the first rule under `User-agent: *` something you wrote on purpose?