Book a strategy call
Book a strategy call

AI crawlers

Reference pages for 16 AI and search crawlers: what each one does, its documented user agent, the robots.txt rule and how to verify it.

Every entry comes from the operator's own documentation, linked on its page. Want to see how your site treats them? Run the AI crawler access check, or read which AI crawlers to allow.

Search engine

Classic search engines. Google says its AI features are rooted in its core Search systems, so Googlebot access matters for AI Overviews too.

Search index

Crawlers that build the index an AI product searches when it answers. Blocking them removes you as a source in that product.

User-triggered fetch

Fetchers that open a page because a person asked a question that needs it. OpenAI and Perplexity say robots.txt may not apply to theirs; Anthropic says Claude-User follows it.

Model training

Crawlers whose content may be used to train models. Blocking them is a training decision, not a search one.

Control token (no crawler)

robots.txt names with no crawler behind them. They set a preference for content another crawler already fetched.

Last reviewed

Not sure which crawlers your site lets in?

The Diagnostic checks crawler access, rendering and the rest of the six method stages, then ranks what to fix first.

Book the DiagnosticRun the free check