Most sites have one robots.txt decision to make about AI, and they make it badly. They see a crawler named after an AI company and block it. Then they wonder why their pages stop appearing in AI search answers.
The fix is to stop treating “AI crawler” as one thing. Each vendor runs separate agents for three different jobs. You can allow some and refuse others. This article shows how, using only what the vendors publish.
The short answer
Allow the search and user-fetch agents if you want to be found and quoted in AI answers. Decide separately whether to allow the training crawlers. Treat those as two policies, not one.
Every vendor below documents this split, and our AI crawler reference has a page for each agent. OpenAI’s crawler page states that “each setting is independent of the others”. It gives the example of allowing OAI-SearchBot for search while disallowing GPTBot for training.
Which AI crawlers are there, and what does each one do?
Three kinds, each with a different job. AI companies send training crawls, search-index crawls and user-triggered fetches to your site. They look similar in logs but serve different purposes.
Training crawlers collect pages that may be used to train future models. Blocking them keeps your content out of training sets. It does not, by itself, remove you from search features.
Search crawlers build the index that powers an AI product’s search or citations. Blocking them means your site is not shown as a source in those answers.
User fetchers retrieve a page because a person asked a question that needs it. These are not scheduled crawls. OpenAI and Perplexity say their user fetchers may not follow robots.txt; Anthropic says its bots, Claude-User included, honour it.
Training
- OpenAI: GPTBot
- Anthropic: ClaudeBot
- Google: Google-Extended (a token, not a crawler)
Blocking keeps content out of future training. Documented as having no effect on search.
Search index
- OpenAI: OAI-SearchBot
- Anthropic: Claude-SearchBot
- Perplexity: PerplexityBot
Blocking removes the site as a source in that product’s answers.
User fetch
- OpenAI: ChatGPT-User
- Anthropic: Claude-User
- Perplexity: Perplexity-User
Started by a person’s question. OpenAI and Perplexity say robots.txt may not apply; Anthropic says its bots honour it.
What each vendor says
OpenAI
OpenAI documents three agents. GPTBot crawls content that may be used to train OpenAI’s models. The page says that “disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models”.
OAI-SearchBot is the search agent. It surfaces sites in ChatGPT’s search features. Sites that opt out “will not be shown in ChatGPT search answers”, though they can still appear as navigational links. OpenAI says robots.txt changes can take about a day to apply.
ChatGPT-User handles actions a person starts inside ChatGPT, such as visiting a page to answer a question. It is not used for automatic crawling. OpenAI notes that because a user initiates these requests, “robots.txt rules may not apply”.
Anthropic
Anthropic’s support article also lists three agents. ClaudeBot collects content that could contribute to model training. Claude-SearchBot indexes the web to improve search results. Claude-User fetches pages when a Claude user asks a question that needs them.
Anthropic says blocking Claude-SearchBot “may reduce visibility in user search results”. Blocking Claude-User prevents retrieval for user queries. Its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”, and the page explicitly advises against blocking by IP address. IP blocks can stop the bot from reading your robots.txt at all.
Perplexity
Perplexity documents two agents. PerplexityBot is “designed to surface and link websites in search results on Perplexity” and “is not used to crawl content for AI foundation models”. Perplexity-User supports user actions within the product. Because a person requested the fetch, this fetcher “generally ignores robots.txt rules”.
Note what that means. Perplexity has no training crawler listed on this page. If you block PerplexityBot, you are blocking search, not training.
Google is the odd one out. Google’s common crawlers page explains that Google-Extended “is a standalone product token” rather than a separate crawler. Googlebot does the crawling. The Google-Extended token controls whether that crawled content may be used to train future Gemini models or for grounding in Gemini Apps and Vertex AI.
Google states that “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. It does not say anything about AI Overviews on that page. Our reading is that AI Overviews and AI Mode are governed by Googlebot and Search Console controls, not Google-Extended. Confirm that against Google’s current documentation before relying on it.
A decision table you can copy
Use this table to write your policy. Pick a row for each question, then translate it into robots.txt rules. The agent names come from the vendor pages linked above.
| Question | If yes, allow | If no, disallow |
|---|---|---|
| Do you want to appear as a source in ChatGPT search? | OAI-SearchBot | OAI-SearchBot |
| Do you want to appear as a source in Claude’s search? | Claude-SearchBot | Claude-SearchBot |
| Do you want to appear as a source in Perplexity? | PerplexityBot | PerplexityBot |
| May your content train OpenAI models? | GPTBot | GPTBot |
| May your content train Anthropic models? | ClaudeBot | ClaudeBot |
| May Google use crawled content to train Gemini or ground Gemini Apps? | Google-Extended | Google-Extended |
| Should people using these assistants be able to open your page on request? | ChatGPT-User, Claude-User, Perplexity-User | Same (but see note on user fetchers) |
The user-fetcher row needs care. OpenAI and Perplexity say their user fetchers may not follow robots.txt, so for them a disallow rule is a stated preference, not a technical block. Anthropic says Claude-User honours it. If you need to stop user fetches, that is a server-side decision, and you should weigh what it costs in lost visibility.
A worked example
Suppose a B2B software company wants to be cited in AI answers but does not want to feed training sets. Its robots.txt would allow OAI-SearchBot, Claude-SearchBot and PerplexityBot. It would disallow GPTBot, ClaudeBot and Google-Extended. It would leave the user fetchers alone.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /This is a starting point, not advice for your situation. A publisher with licensing deals, or a company with legal constraints on data use, may land elsewhere.
How do you check the policy is working?
With three checks, which catch most mistakes.
First, fetch your robots.txt and confirm every named agent is spelt exactly as the vendor documents it. Case and hyphens matter. Our AI crawler access check reads the file and fetches a URL as each agent, so it also shows what your CDN or firewall does.
Second, check your CDN and firewall rules. Anthropic’s article warns that IP blocking can prevent its bots from reading robots.txt. A WAF rule that blocks “AI bots” as a category will override the careful policy you just wrote.
Third, verify who is actually hitting you. Anthropic publishes its crawler IP ranges; the support article links to the list. OpenAI and Perplexity publish similar information on their crawler pages. Compare log entries against those lists before deciding a request is genuine.
How often should you review the policy?
Every quarter. These documents change. Agent names are added, and rules about robots.txt handling shift. Put a reminder in the calendar to re-read the four vendor pages linked above every quarter, and diff your robots.txt against them.
What we saw on our own properties
Four of the sites in our case studies record ChatGPT as a source of visits in GA4. These are people who followed a link from an answer, which only happens if the page could be fetched and shown:
| Site | Sessions from chatgpt.com | All sessions in the export | Share |
|---|---|---|---|
| woodcharcoal.in | 51 | 446 | 11% |
| onlinephotoprinting.co.nz | 23 | 231 | 10% |
| firebrickssource.com | 30 | 323 | 9% |
| shopifywebdeveloper.co.uk | 16 | 201 | 8% |
The numbers are small and the sites are young, so read them as evidence that the channel exists, not as a forecast. GA4 cannot say which agent fetched the page or which answer carried the link; the case studies state those limits in full. What they do show is the cost of the opposite decision: a blanket block on OpenAI’s agents would have removed those visits along with the training crawl.
What we don’t claim
Allowing these crawlers makes your pages eligible to be retrieved. It does not mean any AI product will cite you, and we make no claim about how often that happens. Everything above comes from the vendors’ published documentation as read on 9 October 2026. Where their pages are silent, we have said so.