--- title: "Test Whether AI Crawlers Can Reach Your Site, in Five curl Commands" url: https://hostmy.blog/ai-crawler-test/ date: 2026-09-21 modified: 2026-09-07 lang: en author: "Aditya Sharma" description: "Five commands, about four minutes, and you know whether GPTBot, ClaudeBot and the rest can actually fetch your pages." categories: - "AI Search" image: https://hostmy.blog/wp-content/uploads/2026/09/hmb-card-1884-1024x538.jpg word_count: 1752 --- # Test Whether AI Crawlers Can Reach Your Site, in Five curl Commands Every AI SEO checklist starts in the wrong place. Rewrite your sentences, add schema, generate an llms.txt. None of it matters if the crawler gets a 403 at the door. The good news is that the door test takes about four minutes and needs nothing but curl, which is already on your Mac and on your server. It is the access layer of [the wider five layer audit](/five-layer-ai-seo-audit/), and the layer worth clearing first. Run these five in order. Each one answers a question the next one depends on. Five commands, and what a pass looks like Five commands, and what a pass looks like 1. One crawler, one page expect 200, not 403 2. Every crawler at once one status code per user agent 3. robots.txt as a machine reads it the most specific group wins 4. llms.txt content type text/plain, no nosniff 5. Markdown negotiation Vary: Accept on the response Each command answers one question with a value you can read. None of them needs a plugin, an account or a crawl budget. ## Command 1: can one crawler fetch one page Start with the single most important question. Swap in your own URL. `curl -sS -o /dev/null -w '%{http_code}\n' \ -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \ https://yoursite.com/your-post/` Expected output: `200` That is the whole test. `-o /dev/null` throws the body away, `-w '%{http_code}'` prints just the status. A `403` means something refused the request based on the user agent. A `429` means rate limiting. A `503` usually means a challenge page from a WAF. Anything that is not `200` or a clean `301` stops the crawler cold. Use the full user agent string, not the bare token `GPTBot`. Some bot rules match on the whole string and behave differently against a short one, so a bare token can hand you a false pass. ## Command 2: compare every crawler at once One crawler passing does not mean the rest do. Blocking is usually per rule, not global. Put the agents in a file first: `cat > uas.txt <<'TXT' GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot ChatGPT-User|Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot) OAI-SearchBot|Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot) ClaudeBot|Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Claude-User|Mozilla/5.0 (compatible; Claude-User/1.0; +Claude-User@anthropic.com) PerplexityBot|Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) CCBot|CCBot/2.0 (https://commoncrawl.org/faq/) Amazonbot|Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) meta-externalagent|meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) Bytespider|Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) TXT` Then loop: `URL="https://yoursite.com/your-post/" while IFS='|' read -r name ua; do code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 20 -A "$ua" "$URL") printf '%-20s %s\n' "$name" "$code" done < uas.txt` Here is that loop run against a live post on wordpress.org, September 2026: `GPTBot 200 ChatGPT-User 200 OAI-SearchBot 200 ClaudeBot 200 Claude-User 200 PerplexityBot 200 CCBot 200 Amazonbot 200 meta-externalagent 200 Bytespider 403` Nine open, one refused. Now look at what that same site says in its own robots.txt: `User-agent: Bytespider Allow: /` The file says yes. The server says 403. Both were checked minutes apart and the 403 reproduced on three consecutive runs. That gap is the entire reason this test exists. robots.txt is a request you publish, not a description of what your stack does. A WAF rule, an edge setting or a host level blocklist sits in front of it and wins, and nothing in your SEO tooling will ever mention the disagreement. [Your host quietly refusing crawlers](/host-blocking-ai-crawlers/) is the version of this that survives longest, because nobody on the site ever sees it. Run the loop twice before you act on it. One of the runs while writing this returned a timeout rather than a status for the same agent, which is a network blip rather than a block. A single result is a data point, not a finding. That loop is POSIX shell, so it runs the same under `sh`, `bash` and `zsh`. There are roughly 29 AI crawler agents worth knowing, including cohere-ai, DuckAssistBot, MistralAI-User, AI2Bot, Diffbot, YouBot, Timpibot, Omgilibot and ImagesiftBot. Ten is enough for a first pass. Add the rest to `uas.txt` once you have a baseline. Two names on that list of 29 belong in robots.txt rather than in this loop. Google-Extended and Applebot-Extended are control tokens used to opt out of AI training, so they belong in your robots.txt groups, not in a curl test. Sending one as a user agent tells you nothing. ## Command 3: read robots.txt as a machine reads it `curl -sSIL https://yoursite.com/robots.txt | grep -iE '^(HTTP/|content-type)' curl -sSL https://yoursite.com/robots.txt` Expected output from the first line: `HTTP/2 200 content-type: text/plain; charset=utf-8` Two traps here. The `-L` matters. Without it, a site that redirects `http` to `https`, or bare domain to `www`, hands you a `301` and an empty body, and you conclude the file is missing. Anchoring the grep with `^` matters too. Plain `grep -i content-type` also matches a `vary: accept, content-type` header, so you get two lines and misread which one is the real content type. Now read what the file says, remembering that robots.txt matching is most specific wins rather than additive. A named crawler group replaces the wildcard group entirely. Give GPTBot its own two line block and every `Disallow` you wrote under `User-agent: *` silently stops applying to it. That trap has its own walkthrough in [auditing robots.txt by hand](/audit-robots-txt-by-hand/), and it is the most common self inflicted wound in this whole area. ## Command 4: check llms.txt is served as the right type `curl -sSIL https://yoursite.com/llms.txt \ | grep -iE '^(HTTP/|content-type|x-content-type-options)'` Three live examples, checked September 2026: `https://wordpress.org/llms.txt content-type: text/plain; charset=utf-8 https://yoast.com/llms.txt content-type: text/plain x-content-type-options: nosniff https://platform.claude.com/llms.txt content-type: text/plain; charset=utf-8 x-content-type-options: nosniff` All three correct. The failure mode looks like `content-type: text/html` on a `.txt` file, which happens when a rewrite rule or a cache serves the file through the page handler. That combination is worse than it looks. Serving llms.txt as `text/html` while also sending `X-Content-Type-Options: nosniff` tells a strict client not to guess the type, so it will not parse the file as text. You have published a file that says do not read me. Set expectations honestly on the file itself. SE Ranking looked at roughly 300,000 domains and found no correlation between having llms.txt and anything. Ahrefs checked 137,000 domains, found about 38,000 carrying the file, and 97 percent of those files were never requested once. Google has said it does not use the file. Serve it correctly if you publish it, and put your effort into the other four commands, which sit much closer to [the four checks that decide whether you get used as a source](/ai-seo-checker-curl/). ## Command 5: check Markdown content negotiation A page served as Markdown is dramatically cheaper for a machine to read than one wrapped in a theme. `curl -sSL -H "Accept: text/markdown" -D - -o /dev/null \ https://yoursite.com/your-post/ | grep -iE '^(HTTP/|content-type|vary)'` Run against a live wordpress.org post: `HTTP/2 200 content-type: text/markdown; charset=utf-8 vary: Accept` That is a correct implementation. The server noticed the `Accept` header, returned Markdown, and set `Vary: Accept` so caches key on it. Two honest caveats. `Vary: Accept` is necessary but not sufficient. Cloudflare does not honour `Vary` by default, so an origin doing the right thing can still sit behind an edge that serves one cached copy to everyone. Test through your CDN, not just against the origin. And the audience is narrower than the hype suggests. Markdown negotiation is honoured by coding agents. ChatGPT browse, Claude.ai, Perplexity, Gemini and Grok fetch the HTML, a difference the [request by request path from an answer to your server](/ai-crawlers-request-by-request/) makes concrete. Worth doing, not worth expecting miracles from. To see the body rather than the headers, drop `-D -` and `-o /dev/null`: `curl -sSL -H "Accept: text/markdown" https://yoursite.com/your-post/ | head -20` ## Two follow-up checks worth keeping When a crawler gets a different answer from a browser, the payload size gives it away before the status code does. Ask for both and compare bytes: `for ua in "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/128.0 Safari/537.36" \ "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)"; do curl -sS -o /dev/null -w '%{http_code} %{size_download} bytes\n' -A "$ua" https://yoursite.com/ done` Expected output, when nothing is cloaking: `200 223463 bytes 200 223463 bytes` Two identical numbers mean both visitors got the same page. A much smaller number on the bot line means something is serving a stripped or challenge version, and the `200` was hiding it. Second, follow the redirect chain, because a crawler that hits three hops before the content is spending budget on nothing: `curl -sS -o /dev/null -L \ -w 'final=%{url_effective} redirects=%{num_redirects} code=%{http_code}\n' \ http://yoursite.com/your-post` ``` final=https://yoursite.com/your-post/ redirects=2 code=200 ``` Two hops is http to https and then the trailing slash. Three or more is worth flattening. ## What to do with a failing result A `403` in command 1 or 2 has four usual causes, in the order they turn up: a security plugin with an AI crawler blocklist enabled by default, Cloudflare bot protection at a setting nobody remembers choosing, a hosting level rule applied at the account rather than the site, and a hand written robots.txt group that does the opposite of what its author intended. Test from outside your own network as well. A rule that whitelists your office IP hands you a clean `200` while every crawler gets a challenge page. curl tells you what a crawler could fetch; [reading your access log for the agents that actually arrived](/read-server-log-ai-crawlers/) tells you which ones bothered. Once the door is open, the next question is what the crawler receives after it walks through. That is a different measurement, covered in [a site-wide sweep for unreadable pages](/find-pages-ai-cannot-parse/). ## If you would rather not run scripts Everything above is manual on purpose, because the mechanism is worth understanding once. If you would rather have it checked continuously, RankReady is a WordPress AI SEO plugin that runs these same checks from the admin, writes correct robots.txt groups for all 29 crawlers, serves llms.txt with the right content type, and handles the Markdown endpoint with the cache headers set properly. It runs alongside Rank Math, Yoast, AIOSEO and SEOPress rather than replacing them, because it does a different job: those handle titles, meta and sitemaps, this handles machine access. Setup takes about five minutes. To be precise about what that buys you: it makes your pages reachable and cheap to parse. Nobody can promise you citations, and anyone who does is guessing. Reachability is the part you control, and it is the part most sites get wrong. The plugin is on [WordPress.org](https://wordpress.org/plugins/rankready-ai-llm-seo/). Run command 2 against your own site now. Which of the ten came back with something other than 200, and do you remember choosing that?