How to find AI crawlers in your server logs, and verify them

The short answer. Every request in a standard access log carries the client’s IP address, the time, the request line, the status code and the user agent string. Search the user agent field for the tokens the operators document (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, CCBot), then confirm each match by its IP address: OpenAI, Anthropic, Perplexity, Apple, Google and Common Crawl publish IP ranges, and Google, Bing, Apple and Common Crawl also document a reverse DNS check. A user agent match on its own proves nothing, because any client can send any string. Logs show what crawlers fetched and what you served them; they do not show whether an answer used or cited your page.

Everything below comes from the Apache and nginx logging documentation, the crawler pages of OpenAI, Anthropic, Perplexity, Apple and Common Crawl, the verification pages of Google and Bing, and Cloudflare’s Logpush and Instant Logs pages, as read on 15 October 2026.

What does a log line contain?

Most servers write the Combined Log Format, or something close to it. nginx predefines it and uses it by default (access_log logs/access.log combined;), and Apache documents it as LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined. A line looks like this (synthetic, with a documentation IP address):

192.0.2.10 - - [15/Oct/2026:08:14:02 +0000] "GET /guides/ai-search/ HTTP/1.1" 200 18422 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

Field	What Apache says it is	What it is for in bot analysis
192.0.2.10 (%h)	“the IP address of the client (remote host) which made the request”	The only field you can verify. Behind a proxy or CDN it is the proxy’s address (see below)
15/Oct/2026:08:14:02 +0000 (%t)	“The time that the request was received”	Crawl frequency, first and last seen
"GET /guides/ai-search/ HTTP/1.1"	The request line: method, resource and protocol	Which URLs each agent fetches, including /robots.txt
200 (%>s)	“the status code that the server sends back to the client”	Whether the bot got the page, a redirect, a block (4xx) or an error (5xx)
18422 (%b)	“the size of the object returned to the client, not including the response headers”	Spotting empty or truncated responses
"-" (Referer)	“the site that the client reports having been referred from”	Usually empty for crawlers
"Mozilla/5.0 … GPTBot/1.4 …"	“the identifying information that the client browser reports about itself”	The claimed identity: where the token match happens

Apache adds a caution worth taking literally: “The IP address reported here is not necessarily the address of the machine at which the user is sitting. If a proxy server exists between the user and the server, this address will be the address of the proxy”. If your site sits behind a CDN or load balancer, your origin log may record the CDN’s addresses, and IP verification then needs the client address the CDN passes on, or the CDN’s own logs.

Which user agent tokens should you look for?

Match the token, not the whole string. OpenAI notes its version numbers change (its page now shows GPTBot/1.4), and several operators append extra text.

Token	Operator	What the operator says it does	Reference
GPTBot	OpenAI	Crawls content “that may be used in training our generative AI foundation models”	GPTBot
OAI-SearchBot	OpenAI	“used to surface websites in search results in ChatGPT’s search features”	OAI-SearchBot
ChatGPT-User	OpenAI	User actions; “not used for crawling the web in an automatic fashion”	ChatGPT-User
ClaudeBot	Anthropic	Collects web content that may be used for training	ClaudeBot
Claude-SearchBot	Anthropic	“navigates the web to improve search result quality for users”	Claude-SearchBot
Claude-User	Anthropic	Fetches pages when a person asks Claude a question	Claude-User
PerplexityBot	Perplexity	“designed to surface and link websites in search results on Perplexity”	PerplexityBot
Perplexity-User	Perplexity	“supports user actions within Perplexity”	Perplexity-User
Googlebot	Google	Google says preferences addressed to it “affect Google Search”	Googlebot
Bingbot	Microsoft	Bing’s search crawler	Bingbot
Applebot	Apple	Crawls for “various features, such as the search technology”; may help train Apple models	Applebot
CCBot	Common Crawl	Identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/)	CCBot

Two tokens you will never find in a log: Google says Google-Extended “doesn’t have a separate HTTP request user agent string” (crawling is done with Google’s existing user agents), and Apple says “Applebot-Extended does not crawl webpages.” And OpenAI says that when GPTBot and OAI-SearchBot fetch robots.txt they may add a robots.txt marker to the user agent string “to help site owners distinguish those requests from requests for other resources, especially when logs do not include paths.”

Why is a user agent match not enough, and how do you verify one?

The user agent is a header the client writes about itself. Common Crawl says it is “aware of crawlers falsely identifying themselves as CCBot”, and Bing’s verification post starts from “what appears to be Bingbot traffic in your server logs based on a user agent string”. Verification means checking the IP address against something the operator controls. There are two documented methods, and operators differ in which they offer.

Published IP ranges. Several operators publish JSON files of CIDR ranges; check whether the logged IP falls inside one of them.

• OpenAI: one file per agent, gptbot.json, searchbot.json and chatgpt-user.json.
• Anthropic: a single list at claude.com/crawling/bots.json; Anthropic says a source IP on that list “indicates that the crawler is coming from Anthropic”. The file we read has no per-agent split, so it confirms the operator, not which of the three Claude agents it was.
• Perplexity: perplexitybot.json and perplexity-user.json. Perplexity recommends you “combine both User-Agent string matching and IP address verification” and “Always use the most current IP ranges from the official JSON endpoints.”
• Google: several files by crawler type, including common-crawlers.json for Googlebot and user-triggered-fetchers.json, linked from its verification page.
• Apple: Apple says to “match the IP address with a CIDR prefix contained in” the applebot.json file.
• Common Crawl: IPv4 and IPv6 ranges at ccbot.json.

Every file we opened carries a creationTime field, and the lists change, so fetch them on a schedule rather than copying them once.

Reverse then forward DNS. For operators that document it, look up the hostname of the IP, check the domain, then look up that hostname and confirm it returns the same IP. Google’s steps: run a reverse lookup with host, verify the name is in googlebot.com, google.com or googleusercontent.com, run a forward lookup on that name, and verify it matches “the original accessing IP address from your logs”. Bing’s: the reverse lookup should resolve to a name ending in search.msn.com, and a forward lookup on that name should resolve “back to the same IP address”; Bing Webmaster Tools also has a Verify Bingbot tool that takes an IP address. Apple identifies Applebot by reverse DNS in “the *.applebot.apple.com domain”, and Common Crawl documents reverse DNS for CCBot but says it is not yet supported over IPv6.

Where the docs are silent: OpenAI, Anthropic and Perplexity document IP lists only, not a reverse DNS method. Bing’s pages we read describe reverse DNS and the lookup tool; a bingbot.json file is served at bing.com/toolbox/, but neither page links to it, so we do not rely on it. Apache also advises against resolving hostnames at write time (“not recommended since it can significantly slow the server”) and suggests a post-processor such as logresolve instead: verify afterwards, not in the log pipeline.

1. 1

   Read the line

   IP, time, request line, status, bytes, user agent.

   Behind a CDN, the IP may be the proxy’s

2. 2

   Match the token

   GPTBot, ClaudeBot, PerplexityBot, Googlebot and so on, ignoring version numbers.

   A claim, not proof

3. 3

   Verify the IP

   Inside the operator’s published ranges, or a matching reverse and forward DNS lookup.

   Method depends on the operator

4. 4

   Measure

   Hits, status codes, URLs, robots.txt fetches, errors served to bots.

Not in the log

Whether an AI answer used, quoted or cited the page. Logs end at the response you served.

What should you measure once the hits are verified?

The log tells you what each verified agent asked for and what it received. The useful cuts:

• Hits per agent over time. Which operators crawl you at all, and whether a training crawler, a search crawler or a user-triggered fetcher accounts for most of it. These do different jobs; the access policy guide explains how to treat each.
• Status codes per agent. A crawler that mostly receives 403 or 429 is being blocked, usually by a firewall or bot-protection rule rather than robots.txt. 5xx responses mean it hit errors. Neither is visible from robots.txt.
• Which URLs. Are the pages you care about being fetched, or only the home page and old URLs that redirect?
• robots.txt fetches. Whether each crawler reads /robots.txt and what status it got. A firewall block on the crawler’s IP can stop it reading the file at all; Anthropic gives that as its reason for advising against IP blocking as an opt-out.
• Rendering resources. Whether an agent requests your JavaScript and CSS files or only the HTML. If it fetches HTML alone, what it sees is what is in that HTML; our JavaScript rendering guide covers why that matters and the no-JavaScript content check shows what is in your server HTML.
• Unverified claimants. Requests with a crawler’s token from outside its published ranges. These are not that crawler, whatever they say, and they belong in your bot-protection rules, not your crawl analysis.

Our AI bot log analyser does the first part in your browser: paste lines or choose a file and it classifies requests by documented token, with hits, status-code mix, first and last seen and top paths per agent. It reads strings only and does not resolve or check IP addresses, so its rows are requests that claimed to be each agent; verify the IPs with the methods above.

Origin logs or CDN logs?

If a CDN answers some requests from cache, your origin server never sees them, and the origin log under-counts crawler visits; requests the CDN blocks never reach the origin either. A CDN’s own logs see every request at the edge.

On Cloudflare, what we could verify on its documentation: Logpush, which pushes logs “in near real-time to storage or SIEM”, is listed as available on Free, Pro, Business and Enterprise, though the datasets you can use depend on the products on your account (Workers Trace Events Logpush needs the Workers Paid plan). Instant Logs, which gives “a live stream of the traffic for their domain from the Cloudflare dashboard or from a command-line interface (CLI)”, is listed for Business and Enterprise only, with “Only one active Instant Logs session per zone” and a “Maximum session time” of 60 minutes. We have not checked other CDNs for this article; read your provider’s logging documentation for what it records and on which plans.

What can logs not tell you?

• Whether content was used. A 200 to GPTBot or ClaudeBot shows the page was delivered, not whether it went into a training set, and the operators do not report that per site.
• Citations and mentions. Nothing in a log records an AI answer quoting or linking your page. Measure that from the outside with a sampled prompt set, as in our sampling method, and from the AI reports in Search Console and Bing Webmaster Tools.
• Visitors who came from an answer. Those arrive as ordinary browsers with a referrer; segment them in analytics with the AI referral segment builder.
• Intent behind a user-triggered fetch. A ChatGPT-User, Claude-User or Perplexity-User request shows someone’s question needed the page, not what the question was.

A checklist

1. Confirm your server writes the Combined Log Format, or at least the client IP, time, request line, status and user agent.
2. If you sit behind a CDN or proxy, find out which IP your origin logs record, and use the CDN’s logs if it serves from cache.
3. Match requests on the documented tokens, ignoring version numbers, and keep the robots.txt marker in mind for OpenAI’s crawlers.
4. Verify every match: published IP ranges for OpenAI, Anthropic, Perplexity, Apple, Google and Common Crawl; reverse then forward DNS for Google, Bing, Apple and Common Crawl.
5. Refresh the IP range files on a schedule rather than copying them once.
6. Report per verified agent: hits, status codes, top URLs, robots.txt fetches and any 4xx or 5xx served to it.
7. Check whether the crawlers you allow can fetch the pages you care about, then confirm with the AI crawler access checker.
8. Treat token matches from outside the published ranges as impostors and handle them in bot protection.
9. Re-read the operators’ crawler pages quarterly; tokens, version numbers and IP lists change.

What we don’t claim

• We do not claim that crawler visits lead to rankings, citations or mentions in any AI assistant. A log shows delivery, not use.
• We do not claim the verification methods are uniform. Each operator documents its own, and where an operator documents no method, a user agent match stays unverified.
• We do not claim the details are permanent. Vendor documentation, tokens and IP lists change without notice; re-read the sources before relying on them.