robots.txt

robots.txt is a plain-text file at a site’s root that tells crawlers which paths they may or may not fetch, using User-agent, Allow and Disallow rules. It controls crawling, not indexing: a page blocked in robots.txt can still appear in results if other pages link to it, just without a snippet, and a page you want out of the index needs a noindex directive the crawler is allowed to see.

How AI crawlers use it

The crawler documentation from OpenAI, Anthropic, Perplexity and Google each names its agents and states which ones honour robots.txt. The pattern is consistent: training crawlers (GPTBot, ClaudeBot, Google-Extended as a control token) and search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) obey it; user-triggered fetchers (ChatGPT-User, Claude-User) may fetch a page a person asked for regardless.

Writing a policy

Decide per purpose, not per vendor: do you want training, search indexing and user fetches? Then write one group per agent you want to treat differently and leave * for the rest. Google’s documentation explains precedence and matching. Our AI crawler access tool tests a URL as each documented agent.