How Claude chooses sources: what Anthropic documents, and what it does not

The short answer. Anthropic runs three crawlers and says all of them honour robots.txt. To be eligible as a source when Claude searches the web, a page should be open to Claude-SearchBot and Claude-User, free of noindex, and reachable by the search partners Anthropic says supply its results. Anthropic says every web search answer in Claude “includes citations”. It does not document how sources are chosen or ranked, and it does not name the partner behind text results.

Everything below comes from three Claude Help Center articles, on Anthropic’s crawlers, on web search and on blocking or removing content, plus the web search tool documentation for the Claude API, as read on 15 October 2026.

Which Anthropic agents visit your site?

Three, each with its own robots.txt token.

Agent	What Anthropic says it does	What Anthropic says blocking it changes
ClaudeBot	Collects “web content that could potentially contribute to” the training of Anthropic’s models	Signals that “the site’s future materials should be excluded from our AI model training datasets”
Claude-User	“When individuals ask questions to Claude, it may access websites using a Claude-User agent”	Prevents retrieval “in response to a user query, which may reduce your site’s visibility”
Claude-SearchBot	“Navigates the web to improve search result quality for users”	Prevents indexing “for search optimization, which may reduce your site’s visibility and accuracy”

Two points stand out against the other vendors. First, ClaudeBot is the training crawler, so blocking it is a training decision, separate from search. Second, Anthropic does not carve its user-initiated agent out of robots.txt. OpenAI and Perplexity both describe their user-initiated agents as not bound by robots.txt in the usual way (see our ChatGPT guide and our Perplexity guide); Anthropic’s page says its bots respect “do not crawl” signals “by honoring industry standard directives in robots.txt”, and presents Claude-User as a way for site owners “to control which sites can be accessed”.

Each agent has a reference page: ClaudeBot, Claude-User and Claude-SearchBot.

What does Anthropic ask site owners to do?

Use robots.txt, and only robots.txt, for the crawlers.

1. Set rules per agent in robots.txt. The article’s example is User-agent: ClaudeBot with Disallow: /, placed “in your top-level directory”, and Anthropic asks you to do this “for every subdomain that you wish to opt out from”.
2. Use Crawl-delay if you need to slow it down. Anthropic supports “the non-standard Crawl-delay extension to robots.txt”, for example Crawl-delay: 1.
3. Do not rely on IP blocking to opt out. Anthropic says blocking its IP addresses “may not work correctly”, because doing so “impedes our ability to read your robots.txt file”. It publishes one IP list so you can check that a request claiming to be from Anthropic really is: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.” The list we read is not split by agent.

The practical reading: if your firewall or bot protection blocks unknown crawlers, let Anthropic’s published ranges through and make your choices in robots.txt, where Anthropic says they are honoured.

How does Claude search the web?

In the Claude apps, the help article says that when you ask about topics that benefit from current information, “Claude invokes a search tool to inform and ground its generated responses with content from the live web”, and that “every response includes citations”. Responses include “direct citations to sources”, “source links for further reading” and “relevant quotes when appropriate”. Web search is a setting on Team and Enterprise workspaces that an owner turns on, and users can switch it per chat, although the article notes that in the newer Claude experience there is no toggle and “Claude searches the web when it helps”. With web search on, Claude can also read a URL you give it, which Anthropic calls web fetch.

Where do the results come from? The removal article refers to Anthropic’s “partners” and “search partners”, who index content and send it in response to a search. It names Bing only for images: “Image search is powered by Bing.” Neither article names the provider for text results, so we do not either.

1. 1

   Reachable

   Allowed for Claude-SearchBot and Claude-User in robots.txt.

   Stopped by: a Disallow for either agent

2. 2

   Indexed

   Indexed by Claude-SearchBot and by Anthropic’s search partners.

   Stopped by: noindex, removal or a password

3. 3

   Retrieved

   Returned by a search Claude runs for the question, or fetched as Claude-User.

4. 4

   Cited

   Shown as a citation with a source link.

   Not documented: how sources are chosen

Separately: ClaudeBot

Collects content that may be used for model training. A training choice, not a search one.

How do you keep a page out of Claude’s answers?

The removal article lists the options for site owners who want to control “what shows up in Claude outputs that use web search”:

• Remove the content from your site, which Anthropic calls “the best way”.
• Password-protect it.
• Add noindex. Anthropic describes it as a rule “that tells our partners not to index your content so that they don’t send it to us”. The page can still be visited directly, “but the content will not appear in Claude outputs that use web search”.
• Disallow images and video in robots.txt, because “our search partners only index images and videos that their bots are allowed to crawl”.
• Disallow Anthropic’s bots, as above.
• Ask for a URL to be blocked if it already appears in Claude outputs, by email to Anthropic with proof that you own the domain.

The same list tells you what to avoid on pages you do want used: a stray noindex, a login wall, or a Disallow for Claude-SearchBot or Claude-User.

Is the Claude API the same as the Claude apps?

No, and it is worth keeping them apart. The web search tool is what developers add to their own applications built on the Claude API. It documents behaviour the consumer help article does not:

• “Claude determines when to search based on the prompt”, the API runs the searches, and Claude ends its turn “with cited sources”.
• “Citations are always enabled for web search.” Each citation carries the source’s URL, its title and “up to 150 characters of the cited content”.
• Developers can pass allowed_domains or blocked_domains, “not both”, to limit which sites are searched, and organisation administrators can restrict domains in the Claude Console.
• Newer versions of the tool can filter results with code before they reach Claude, keeping “only relevant information”.

So a product built on the API may only ever search a short allowlist of domains, and nothing you do on your site changes that. The API page also says that when developers show API output to end users, “citations must be included to the original source”. It does not explain how results are ranked or which provider runs the searches.

What does Anthropic not document?

• Selection and ranking. None of the pages linked here explains how Claude chooses which sources to cite or in what order.
• The text search provider. Anthropic refers to search partners and names Bing only for image results.
• A site-level report. Anthropic offers site owners no equivalent of Search Console’s Generative AI performance report or Bing’s AI Performance report.
• Referral tagging. The pages we read say nothing about tagging links from Claude with a UTM parameter, unlike OpenAI’s utm_source=chatgpt.com.
• Special markup or files. Nothing asks for structured data, llms.txt or a dedicated sitemap.
• Per-agent IP ranges. One list covers all Anthropic crawlers.

Measure it from the outside instead. Analytics can count visits that arrive with claude.ai as the referrer; our AI referral segment tool includes it by default. Many app visits arrive without a referrer, so treat the count as a floor. None of our published case studies reports a Claude referral figure, so we have none to share. Claude is not one of the engines in our current Lab prompt set, but the sampling method works the same way for it if you run it yourself.

A checklist

1. robots.txt allows Claude-SearchBot and Claude-User (check the live file, and every subdomain you care about).
2. You have decided on ClaudeBot separately, as a training question.
3. Your firewall or bot protection lets Anthropic’s published IP ranges through, so robots.txt can be read and your rules applied.
4. Pages you want in answers have no noindex and no login wall; pages you want out have noindex or are removed.
5. Analytics groups claude.ai referrals into a report you check monthly, knowing it undercounts.
6. If Claude matters to your buyers, add it to a sampled prompt set and score mentions, citations and accuracy.
7. Calendar a quarterly re-read of the three help articles; Anthropic offers an update subscription on the crawler article.

Our AI crawler access tool fetches a URL as each of Anthropic’s agents and reports what each receives.

What we don’t claim

• We do not claim to know how Claude selects or ranks sources. Anthropic does not document it in the pages linked here.
• We do not claim to know which provider supplies Claude’s text search results. Anthropic names one only for images.
• We do not claim that any step above causes a mention or a citation. They make a page eligible to be used.
• We do not claim the details are permanent. Vendor documentation changes without notice; re-read the sources before relying on them.