Search index · Anthropic
Claude-SearchBot
Claude-SearchBot is Anthropic's crawler for its AI search index. It honours robots.txt.
- Operator
- Anthropic
- Purpose
- Search index
- robots.txt
- Honoured
- Last checked
Documented user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.comMatch on the token Claude-SearchBot rather than the whole string, since the version number inside the string can change. A user agent alone can be forged, so verify it as shown below.
robots.txt rules
To block it everywhere:
User-agent: Claude-SearchBot
Disallow: /To allow it explicitly, for example when a broader rule blocks other agents:
User-agent: Claude-SearchBot
Allow: /What blocking it changes
Anthropic says blocking it prevents its system from indexing your content for search, which may reduce your visibility and accuracy in Claude’s search responses.
- It is separate from ClaudeBot: allowing it while blocking ClaudeBot keeps search visibility without opting into training.
How to verify a request is genuine
Match the requesting IP against the crawler addresses Anthropic publishes. Source: Anthropic: crawler IP addresses.
Other Anthropic agents
ClaudeBot: model trainingClaude-User: user-triggered fetch
Sources
Anthropic's documentation for Claude-SearchBot, read on 15 Oct 2026. Operators change these pages without notice; check before relying on a detail.
Check how your site treats Claude-SearchBot
The AI crawler access check reads your robots.txt and fetches a URL as this agent, so you see what a CDN or firewall does as well.
Run the free check All AI crawlers