ClaudeBot

ClaudeBot is Anthropic’s crawler for collecting web content that may be used to train Claude models. It identifies itself with the token ClaudeBot, and Anthropic’s documentation states that it respects robots.txt.

The three Anthropic agents

Anthropic documents three agents with different purposes:

• ClaudeBot: training data collection.
• Claude-SearchBot: builds the index that Claude’s search feature draws on.
• Claude-User: fetches a page when a Claude user asks about it.

A robots.txt rule for ClaudeBot alone does not affect the other two, so a site that wants to be reachable in Claude’s search while opting out of training should write separate groups.

Verifying it

A user-agent string can be forged. Anthropic publishes the IP ranges its crawlers use, so a log line claiming to be ClaudeBot can be checked against them. Verify before drawing conclusions from a log.

The AI crawler access tool tests a URL as each of the three agents.

For the documented user-agent string, the robots.txt rules and how to verify a request, see the ClaudeBot reference page.