Model training · Anthropic
ClaudeBot
ClaudeBot is Anthropic's crawler for content that may be used in model training. It honours robots.txt.
- Operator
- Anthropic
- Purpose
- Model training
- robots.txt
- Honoured
- Last checked
Documented user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Match on the token ClaudeBot rather than the whole string, since the version number inside the string can change. A user agent alone can be forged, so verify it as shown below.
robots.txt rules
To block it everywhere:
User-agent: ClaudeBot
Disallow: /To allow it explicitly, for example when a broader rule blocks other agents:
User-agent: ClaudeBot
Allow: /What blocking it changes
Anthropic says blocking ClaudeBot signals that the site’s future materials should be excluded from AI model training datasets.
- Anthropic supports the non-standard Crawl-delay directive for its bots.
- Anthropic advises against blocking its IP addresses: an IP block can stop the bot reading robots.txt, so the opt-out may not hold.
- Rules must be repeated for every subdomain you want to opt out.
How to verify a request is genuine
Match the requesting IP against the crawler addresses Anthropic publishes. Source: Anthropic: crawler IP addresses.
Other Anthropic agents
Claude-SearchBot: search indexClaude-User: user-triggered fetch
Sources
Anthropic's documentation for ClaudeBot, read on 15 Oct 2026. Operators change these pages without notice; check before relying on a detail. Definition: ClaudeBot in our glossary.
Check how your site treats ClaudeBot
The AI crawler access check reads your robots.txt and fetches a URL as this agent, so you see what a CDN or firewall does as well.
Run the free check All AI crawlers