--- title: "OAI-SearchBot: The Crawler Behind ChatGPT Search" url: https://hostmy.blog/oai-searchbot-wordpress/ date: 2026-10-17 modified: 2026-10-17 lang: en author: "Aditya Sharma" description: "OAI-SearchBot is the crawler that decides whether ChatGPT search can show your pages, and it is separate from GPTBot. Here is how to spot it and let it in on WordPress." categories: - "RankReady" image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2547-1024x538.jpg word_count: 1399 --- # OAI-SearchBot: The Crawler Behind ChatGPT Search OAI-SearchBot is the crawler OpenAI uses to find pages for ChatGPT's search features. It is not the crawler that collects training data, and it is not the agent that fetches a page when a person asks about it. Those are two other agents, and each one is controlled separately. That separation is the whole reason to know the name. Whether ChatGPT search can surface your pages depends on OAI-SearchBot, and on nothing you decide about GPTBot. This post covers what OpenAI says OAI-SearchBot does, how to tell whether it has visited, and what to set on a WordPress site. ## What OpenAI says OAI-SearchBot is for OpenAI's [crawler documentation](https://developers.openai.com/api/docs/bots) is short and specific. OAI-SearchBot is for search: it is used to surface websites in search results in ChatGPT's search features. GPTBot is the separate agent whose robots.txt setting relates to training OpenAI's generative AI foundation models. OpenAI states that each setting is independent of the others, and gives the example of allowing OAI-SearchBot so a site can appear in search results while disallowing GPTBot. Three details on that page are worth knowing. First, a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as navigational links. Second, if you allow both bots, OpenAI says it may use the results of one crawl for both purposes, to avoid crawling twice. Third, a robots.txt change can take about 24 hours to take effect for search, so plan around that before you decide a new rule is not working. OpenAI also runs ChatGPT-User, which fetches a page when a person asks ChatGPT or a custom GPT about it. That is not automatic crawling, and OpenAI says robots.txt rules may not apply to it. The decision about GPTBot depends on how you earn from your content, and it is covered in [our GPTBot post](/gptbot-wordpress/). This post is about the other one. ## How to tell whether it has visited OAI-SearchBot names itself in the user-agent string, and OpenAI publishes the IP ranges it uses at openai.com/searchbot.json. Anyone can fake a user agent, so a log line with the right name proves little by itself. The published ranges are what separate the real crawler from an imitation. To find visits, search your access log for the name and read the status codes, using [our server-log method](/read-server-log-ai-crawlers/). One command is enough to start. `grep -i "OAI-SearchBot" /var/log/nginx/access.log | tail -20` That path is for nginx. On Apache it is usually `/var/log/apache2/access.log`, and on shared hosting the Raw Access or Metrics area in cPanel has the log. The post linked above covers finding it. You want to see 200 responses on real posts, and fetches of robots.txt. A run of 403 responses means something is refusing it, and nothing in WordPress will tell you. ## What the status codes are telling you A visit is only half the answer. The status code says what happened to it. | Status | What it usually means | Where to look | | ------ | --------------------- | ------------- | | 200 | The page was served | Nothing to fix | | A redirect to a login or challenge page | Can be a bot challenge; a plain http to https or www redirect is normal | Follow it with `curl -IL`, then check security plugin, host firewall and CDN bot settings | | 403 | Something refused it by name, rule or IP address | Security plugin rules, host firewall, CDN | | 429 | It was rate limited | CDN or host rate-limit rules | | 500 to 504 | Your server failed, or a WAF challenge (a 503 is often one) | Server errors and the CDN or WAF log, as in the uptime post below | The pattern that matters is a status that differs from what a browser gets. If a browser receives a 200 and the crawler receives a 403, the refusal is aimed at the crawler, and it is rarely something you chose on purpose. Server errors have their own causes, which [our uptime-from-the-logs post](/wordpress-downtime/) walks through. ## Test it from the outside You can check what a request that looks like OAI-SearchBot receives by using the user-agent string from OpenAI's documentation. OpenAI notes that the version number may change. Its robots.txt fetches carry an extra robots.txt marker in the string, which also helps you spot them in a log. `curl -I -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://yourdomain.com/your-best-post/` A 200 is what you want. A 403, or a redirect to a login or challenge page, means a rule in your security plugin, host or CDN is refusing that user agent. The test has one limit. It comes from your own IP address, so it cannot show a block that applies only to OpenAI's published ranges. For that, check your firewall or CDN settings for those ranges. Hosts and CDNs refuse AI crawlers more often than owners expect, as covered in our posts on [hosts that block AI crawlers](/host-blocking-ai-crawlers/) and [Cloudflare's crawler default](/cloudflare-crawler-default-wordpress/). ## Letting it in on WordPress OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. A request can be refused at several layers, and all of them have to agree. A permissive robots.txt does nothing if a firewall refuses the crawler first. | Layer | Who decides | Where to look on a WordPress site | | ----- | ----------- | --------------------------------- | | robots.txt | The crawler chooses to obey it | Your SEO plugin's editor, RankReady, or the file in the web root | | Security plugin or firewall rule | Your site refuses the request | The plugin's firewall and bot settings | | Host firewall | Your host refuses it before WordPress runs | The host dashboard, or a support request | | CDN bot rules | The edge refuses it before it reaches you | The CDN's bot and crawler settings | In robots.txt it looks like this. A named group replaces your global rules for that crawler, so they are repeated inside it. `# Allows OAI-SearchBot everywhere except wp-admin, with the global rules repeated User-agent: OAI-SearchBot Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php` The trap is explained in [our robots.txt audit](/audit-robots-txt-by-hand/), and the file as a whole in [our WordPress robots.txt guide](/wordpress-robots-txt/). Wait about 24 hours after a change before you test it. ## Is it worth letting in? For most sites, yes, and it is the lower-risk half of the AI crawler decision. Allowing OAI-SearchBot concerns whether your pages can appear in ChatGPT's search answers. It is a separate setting from training use, which OpenAI says is governed by GPTBot, so you can allow one and block the other. Blocking it does not fully remove you: OpenAI says opted-out sites can still appear as navigational links, just not in the answers themselves. The way to judge whether it matters for you is to measure it. Readers who arrive from a ChatGPT answer show up in your analytics with chatgpt.com as the source, and a plugin that reads the referrer on the server can count them without any script. If that number is zero after a few weeks of being allowed, the access question is settled and the rest is about what you publish. ## What allowing it does and does not buy you Allowing OAI-SearchBot makes your pages eligible to appear in ChatGPT search results. It does not make them appear, and it does not decide how they are ranked or quoted. Being let in is the first of several steps. What gets cited is a separate question, covered in [our ChatGPT SEO post](/chatgpt-seo-what-changes/), and the wider picture of Google against the other assistants is in [GEO vs SEO on WordPress](/geo-vs-seo-wordpress/). A visit is also not a citation. A crawler fetching a page shows it was read, not that it was quoted. Treat the log as a leading indicator, and test the outcome by asking ChatGPT the question your page answers. ## Where RankReady fits RankReady is a free WordPress plugin that lists OAI-SearchBot among the 29 AI crawlers you can allow or block one by one, and writes the robots.txt rules with your global rules repeated inside them. It also keeps its own log of AI crawler visits and tags each one by intent: training, search indexing or a live fetch. OAI-SearchBot falls under search indexing, which makes it easy to tell apart from GPTBot. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress. It cannot promise that ChatGPT will cite you, and no plugin can. To see whether OAI-SearchBot can reach you today, the free [AI Readiness Checker](https://hostmy.blog/plugins/rankready/ai-readiness-checker/) runs 41 checks against your live site, and needs no account to see the score. The plugin itself is on [WordPress.org](https://wordpress.org/plugins/rankready-ai-llm-seo/). Open your access log and search for OAI-SearchBot. Is the last status code you see a 200, or have you been turned away without knowing?