GPTBot

GPTBot is OpenAI’s web crawler for collecting content that may be used to train its models. It identifies itself with the token GPTBot in its user-agent string and, according to OpenAI’s documentation, honours robots.txt rules.

What it is not

GPTBot is not the crawler behind ChatGPT search results. OpenAI documents separate agents for that: OAI-SearchBot builds the search index, and ChatGPT-User fetches a page when a person asks about it. Blocking GPTBot therefore limits training use without, by OpenAI’s description, removing a site from ChatGPT search. Blocking OAI-SearchBot does the opposite.

Deciding whether to allow it

That is a policy question for the site owner. Allowing it means the content may inform future models; blocking it has no documented effect on search visibility. We state the trade-off and leave the choice to the client.

User-agent: GPTBot
Disallow: /

OpenAI publishes the IP ranges its crawlers use, so a log entry claiming to be GPTBot can be verified. Our AI bot log analyser separates AI agents in a server log by token.

For the documented user-agent string, the robots.txt rules and how to verify a request, see the GPTBot reference page.