--- title: "GPTBot: Should You Allow It, and Has It Even Visited?" url: https://hostmy.blog/gptbot-wordpress/ date: 2026-09-16 modified: 2026-09-03 lang: en author: "Aditya Sharma" description: "GPTBot is one of three OpenAI agents. Here is what each does and how to check whether yours are blocked." categories: - "AI Search" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1871-1024x538.jpg word_count: 1541 --- # GPTBot: Should You Allow It, and Has It Even Visited? A site can return 403 to GPTBot for months before anyone notices. Usually it is not a decision. A security plugin's default rule, quietly refusing anything that did not look like a browser, on a site whose owner had spent that same period asking me why he never appeared in AI answers. The fix took four minutes. Finding it took an hour, because nothing in WordPress tells you that a crawler was turned away. There is no notice, no log in the admin, no email. The refusal happens at the edge and the dashboard stays green, one broken link in [the request chain behind an AI answer](/ai-crawlers-request-by-request/). That is the real GPTBot problem for most WordPress owners. Whether to allow it is a five minute decision. Knowing what your server is currently doing is the part almost nobody has checked. ## GPTBot is one of three agents, and blocking it blocks the least useful one The mistake I see most often is treating OpenAI as a single crawler. It is not, and the distinction changes the answer. | Agent | What it is doing | Blocking it means | | ----- | ---------------- | ----------------- | | GPTBot | Bulk crawling for model training | Your content is less likely to be in future training data | | OAI-SearchBot | Building the search index behind ChatGPT search | Your pages are less likely to be retrievable when someone searches | | ChatGPT-User | Fetching a page because a user asked for it, right now | A user who asked about your page gets nothing back | Check OpenAI's own published list before you write rules, because these names change and a stale robots.txt is worse than none. The practical point is that the three do different jobs, and most people who say "block GPTBot" mean the training one while writing a rule that catches all three. Blocking ChatGPT-User is the one to think hardest about, because that is a real person, right now, who explicitly asked about your page. What 'block GPTBot' actually blocks What 'block GPTBot' actually blocks GPTBot bulk collection for future training OAI-SearchBot builds the index ChatGPT searches at answer time ChatGPT-User one person asked about your page, right now Blocking the first is a rights decision. Blocking the other two removes you from the answer path. GPTBot is the training crawler. The two agents that decide whether you can be used at answer time are the two the rule usually misses. ## Whether to allow it depends on how you make money, not on principle There is a genuine argument for blocking, and pretending otherwise is dishonest. Your work is used, you are not paid, and the citation, if it comes at all, arrives without a click. The zero click arithmetic is unpleasant and laid out in [the cost side of zero click search](/zero-click-search-cost/). Here is how I actually advise clients, split by business model. **Allow, in most cases.** If you sell a product, a service, software, or anything where being found and named leads to revenue, blocking removes you from a channel that is growing while giving you nothing back. You do not get paid for being excluded either. **Think harder.** If display advertising is the whole business, a citation without a click is a straight loss. Blocking is still unlikely to bring the reader back, because the answer will be assembled from someone else's page. What it does is guarantee that the someone else is a competitor. **Block, reasonably.** Paywalled archives, licensed material you do not own outright, member-only content, and anything with a contractual restriction. That is a rights question, not a marketing one. **Do not block by accident.** This is the category most sites are actually in, and it is not a decision at all. My own position, and it took me a year to arrive at it: allow the retrieval agents, decide about the training agent based on how you feel about it, and never let a plugin default make the call for you. Both sides of that argument, stated fairly, are in [the case for blocking and the case for letting them in](/should-you-block-ai-crawlers/). ## Three commands that tell you the truth Your browser will lie to you here. It is logged in, it is cached, and it looks like a browser. Use a terminal. **Check whether you respond at all:** `curl -I -A "GPTBot/1.4" https://yourdomain.com/your-best-post/` A 200 is fine. A 403 means something in your stack is refusing that user agent, and often the rule belongs to [a host refusing crawlers rather than to anything you wrote](/host-blocking-ai-crawlers/). A 301 or 302 to a login or challenge page is the same refusal wearing a nicer suit. **Check what your robots.txt actually says:** `curl -s https://yourdomain.com/robots.txt` Read the whole file. The problem is rarely a line saying GPTBot. It is usually a wildcard group written years ago, or a plugin appending rules underneath the ones you can see in the WordPress settings screen. **Check whether it has ever visited:** `grep -i "gptbot" /var/log/nginx/access.log | tail -20` On Apache the path is usually `/var/log/apache2/access.log`. On managed hosting the log may live somewhere else, and your host will tell you where. Three outcomes, three meanings: - Lines with a 200 status: it is visiting and being served. - Lines with a 403: it is visiting and being refused. This is the eight month problem, and it is the most common finding on the sites my team audits. - No lines at all: it has not been, which is a different problem and usually a discoverability one rather than a blocking one. The full method for reading these logs, including how to count visits per agent and how to spot fake ones claiming to be GPTBot, is in [pulling AI crawler hits out of a raw log](/read-server-log-ai-crawlers/). ## The rule that says what you mean If you have decided to allow retrieval and disallow training, the file is short. Put it in robots.txt at the root. `User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /` Two warnings from experience. First, group matching is per user agent and the most specific matching group wins. A site with a permissive wildcard group and a specific GPTBot group will apply the GPTBot group to GPTBot, and the wildcard to everyone else. Sites get this wrong constantly, usually by writing rules under the wildcard group and expecting them to apply to a named agent that has its own group elsewhere in the file. The same trap applies across [robots.txt rules for the 29 named AI crawlers](/block-ai-crawlers-robots-txt/), not just OpenAI's three. Second, robots.txt is a request, not a wall. Well behaved agents obey it. Anything scraping you under a fake user agent will ignore it entirely, which is why the log check matters more than the file. ## Blocking is not what stops you appearing Worth being blunt about, because there is a common assumption underneath the whole GPTBot question. Unblocking a crawler makes your page fetchable. It does not make it quotable, and it certainly does not make it quoted. In an analysis of 11,346 cited sentences a May 2026 study could extract, the mean cited sentence was 9.27 words and nothing above 18 words was cited even once. 41.9 percent of citations came from the first 30 percent of the page. So access is the entry requirement, and the writing is what does the work after that. A perfectly crawlable page whose claims sit inside 28 word sentences gives a model nothing to lift. The reverse is also true and is why this post exists. Perfect writing on a page returning 403 is a wasted afternoon. The four concrete jobs, in order, are in [AI search optimization translated into WordPress jobs](/ai-search-optimization-wordpress/). ## Doing this in WordPress without editing files Most WordPress owners cannot safely hand-edit robots.txt, because WordPress generates a virtual one, plugins append to it, and any file you drop at the root may or may not win depending on your server. That is the specific job [RankReady](https://wordpress.org/plugins/rankready-ai-llm-seo/) handles. It is a WordPress AI SEO plugin with per-crawler control, so you tick which agents you allow rather than guessing at group syntax, and it also covers the readable side: summary blocks near the top of a post, FAQ and schema output, a Markdown version of each page for machine readers, and freshness signals. It runs alongside Rank Math, Yoast, AIOSEO and SEOPress instead of replacing them, so nothing about your existing setup has to change. Setup takes about five minutes, and the crawler decisions are most of that. What it changes is whether an agent can reach and parse your content. Whether it then quotes you is out of everyone's hands, mine included. ## Do the check before the decision The order matters, because it is easy to spend an evening deciding a policy you are already accidentally enforcing. - Run the curl check on three URLs, including your homepage and two posts. - Read your robots.txt in full, in a terminal, not in a settings screen. - Grep your access log for the agents by name. - Only then decide what you want your policy to be. - Write the policy, then re-run step one to confirm the file does what you meant. Step five is the one people skip and it is where half the mistakes survive. Have you ever actually looked at your own access log, or are you assuming the crawlers are getting in because nothing has told you otherwise?