--- title: "Content Signals in robots.txt explained" url: https://hostmy.blog/docs/rankready/content-signals-robots-txt/ date: 2026-08-24 modified: 2026-08-24 lang: en author: "Aditya Sharma" description: "What the ai-train, search and ai-input directives mean, how to set them in RankReady, and how to check they are in your robots.txt." word_count: 828 --- # Content Signals in robots.txt explained This page explains the three Content Signals directives RankReady can write into your `robots.txt`, how to decide on each one, and how to confirm they are actually being served. ## The gap Content Signals fills Classic `robots.txt` answers one question: may this crawler fetch this path? Allow or disallow, nothing else. That was adequate when fetching and indexing meant much the same thing. It no longer covers what most site owners actually want to say. Plenty of publishers are happy for their pages to be fetched and listed, but not happy for the same content to be used for model training. The old file has no way to express that. Your only options are yes to everything or no to everything. Content Signals adds directives that describe permitted *uses* alongside the existing access rules, so you can allow the fetch and still state a limit on what happens afterwards. ## The three directives RankReady supports the three signals defined by the proposal: | Directive | The use it refers to | | --------- | -------------------- | | `search` | Building a search index and linking back to your page | | `ai-input` | Using your content as input to a generated answer at the time a question is asked | | `ai-train` | Using your content to train or fine tune a model | They are independent. A common position is yes to `search`, yes to `ai-input`, no to `ai-train`: be findable, be usable in an answer that can point back at you, but do not become training material. Another entirely reasonable position is yes to `search` only. There is no default that is correct for everyone, which is why the plugin asks rather than assuming. ## Be honest about what they are Content Signals are a declaration, not a technical control. Like the rest of `robots.txt`, they are read voluntarily. Nothing about the file stops a request being made or governs what happens to your content once it has been fetched. Adoption across operators is uneven and still developing. What you get is a clear, machine readable statement of your position in the standard location, published under your own domain. That has value on its own, and it is worth ten minutes of your time. It is not enforcement, and you should not plan as though it is. ## Set them in RankReady - In wp-admin, open **RankReady**. - Go to the **AI Visibility** tab. The address ends in `tab=crawlers`. - Open the **Robots** subtab, `sub=robots`. - Set your position for `search`, `ai-input` and `ai-train`. - Save. RankReady writes the result into the managed block in your `robots.txt`, between `# BEGIN RankReady` and `# END RankReady`. Anything outside those markers is left alone, including rules from your SEO plugin or your host. Do not hand edit inside the markers, because the block is regenerated from your settings each time they change. ![RankReady robots.txt controls showing AI crawler settings and Content Signals directives](https://hostmy.blog/wp-content/uploads/2026/08/rr-robots-crawlers-v2-scaled.png)Content Signals sit alongside the per crawler controls in the Robots subtab. ## Decide the signals and the crawler list together These two settings interact, and it is easy to end up with a file that contradicts itself. If you disallow a crawler outright, a signal about permitted use is beside the point for that crawler, since you have already refused the fetch. Signals matter most for crawlers you allow. So work in this order. First decide who may fetch, across the 29 named crawlers RankReady covers. Then decide what the content may be used for. A policy assembled that way is one you can explain to somebody later, which is the real test. ## Verify it worked Read the live file rather than the settings screen: `curl -s https://your-site.com/robots.txt` Confirm three things: - The `# BEGIN RankReady` and `# END RankReady` markers are present. - The Content Signals directives appear inside that block with the values you chose. - Your existing rules outside the markers are unchanged. To pull just the signal lines: `curl -s https://your-site.com/robots.txt | grep -iE 'content-signal|ai-train|ai-input|search'` Read the output in context rather than counting matches. The word `search` appears in other places in a robots file, so check that what you are looking at sits inside the RankReady block. To view the managed block on its own: `curl -s https://your-site.com/robots.txt | sed -n '/# BEGIN RankReady/,/# END RankReady/p'` That prints exactly what RankReady is responsible for, which is the part you can change from the settings screen. ## If your changes do not appear - Check for a physical `robots.txt` file in your web root. A real file overrides anything WordPress generates. Rename it and retry. - Clear your page cache and CDN, since robots.txt is cached aggressively and an old copy will be served long after you save. - Go to **Settings, Permalinks** and click Save Changes to rebuild rewrite rules. - Check whether another plugin is also writing to robots.txt. RankReady stays inside its markers, but not every plugin does. ## Review it occasionally Crawler names change, new agents appear, and the signals proposal is still developing. Re-open the Robots subtab when you update RankReady and confirm the settings still say what you mean. It takes a minute and it stops a stale policy sitting on your domain for years.