Search engine · Google
Googlebot
Googlebot is Google's search engine crawler. It honours robots.txt.
- Operator
- Purpose
- Search engine
- robots.txt
- Honoured
- Last checked
Documented user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Match on the token Googlebot rather than the whole string, since the version number inside the string can change. A user agent alone can be forged, so verify it as shown below.
robots.txt rules
To block it everywhere:
User-agent: Googlebot
Disallow: /To allow it explicitly, for example when a broader rule blocks other agents:
User-agent: Googlebot
Allow: /What blocking it changes
Google Search stops crawling the blocked paths. A blocked URL can still appear in results if other pages link to it, but without a snippet, and therefore without being usable in AI Overviews or AI Mode, which follow the normal Search eligibility rules.
- robots.txt controls crawling, not indexing: to keep a page out of Google, let Googlebot crawl it and serve noindex.
- Google renders JavaScript in a second pass, so content that only appears after scripts run is indexed later.
How to verify a request is genuine
Run a reverse DNS lookup on the IP and confirm the host ends in googlebot.com, google.com or googleusercontent.com, then a forward lookup that returns the same IP. Or match the IP against Google’s published crawler ranges. Source: Google: verifying Googlebot.
Other Google agents
Google-Extended: control token (no crawler)
Sources
Google's documentation for Googlebot, read on 15 Oct 2026. Operators change these pages without notice; check before relying on a detail.
Check how your site treats Googlebot
The AI crawler access check reads your robots.txt and fetches a URL as this agent, so you see what a CDN or firewall does as well.
Run the free check All AI crawlers