Model training · Common Crawl
CCBot
CCBot is Common Crawl's crawler for content that may be used in model training. It honours robots.txt.
- Operator
- Common Crawl
- Purpose
- Model training
- robots.txt
- Honoured
- Last checked
Documented user agent
CCBot/2.0 (https://commoncrawl.org/faq/)Match on the token CCBot rather than the whole string, since the version number inside the string can change. A user agent alone can be forged, so verify it as shown below.
robots.txt rules
To block it everywhere:
User-agent: CCBot
Disallow: /To allow it explicitly, for example when a broader rule blocks other agents:
User-agent: CCBot
Allow: /What blocking it changes
CCBot is Common Crawl’s crawler. Common Crawl publishes an open repository of web crawl data that anyone can use, so blocking it keeps future pages out of that public dataset rather than out of one specific product.
- Common Crawl warns that some crawlers falsely identify themselves as CCBot.
- IPv6 addresses do not yet support reverse DNS, per Common Crawl.
How to verify a request is genuine
Run a reverse DNS lookup on the IP (Common Crawl’s hosts are under crawl.commoncrawl.org), then a forward lookup that returns the same IP, or match the IP against Common Crawl’s published ranges. Source: Common Crawl: CCBot IP ranges.
Sources
Common Crawl's documentation for CCBot, read on 15 Oct 2026. Operators change these pages without notice; check before relying on a detail.
Check how your site treats CCBot
The AI crawler access check reads your robots.txt and fetches a URL as this agent, so you see what a CDN or firewall does as well.
Run the free check All AI crawlers