AI search optimization
Be found and described accurately by assistants
- Access
Stage 1: Access
AI bot log analyser
Which AI crawlers hit your server, how often, with which status codes and on which paths, from a log you paste. Runs in your browser; nothing is uploaded.
Free to use. No login, nothing stored.
What it checks
Access- Requests classified by the user agent tokens of the agents in our verified list, plus other known bot tokens
- Hits, status-code mix (2xx to 5xx) and first and last seen per agent
- Top 20 paths per agent
- Agents that are mostly blocked or erroring (4xx and 5xx)
- robots.txt lines for the agents you choose to stop
Method question for this stage: Can crawlers reach and render every key page?
Run the check
Results
No log analysed yet
- Info
One table row per detected agent, then its top paths, then the checks
Operator, purpose, hits, share of requests, 2xx to 5xx counts and first and last seen per agent; a Warning for any agent that is mostly blocked or erroring; counts of skipped lines, requests without a user agent, and everything that is not a known bot.
How we check
The rules this tool tests, and where they come from
A result is a check against a published rule, not a prediction. Every row shows its working.
Rules and sources
- Google: verifying Googlebot and other Google crawlers
- OpenAI: overview of OpenAI crawlers
- Anthropic: does Anthropic crawl data from the web
- Apache: combined log format
Links go to the operator's or standards body's own documentation. We do not link to third-party summaries.
Read more
Next step
Fix what it finds
This check belongs to the Access stage of the Crawl-to-Citation Method. The service below covers the fix; the Diagnostic ranks it against everything else.
Most engagements start with the Search Visibility Diagnostic ($2,500).
It covers all six method stages, so what this tool finds is ranked against everything else before you commit to any work.
Book the diagnosticLog questions
Where logs come from, what a user agent string proves, and the trade-offs of blocking.
Still have questions? Book a callWhere do I get my server logs?
From whatever answers requests for your site. On a server you manage, Nginx writes an access log in the combined format unless configured otherwise, and Apache can be set to the same format; both are plain text files you can copy a slice of. Hosting control panels usually offer a raw access log download. If your site sits behind Cloudflare, Logpush can deliver request logs as CSV or JSON lines, and the field names it uses (ClientRequestUserAgent, ClientRequestURI, EdgeResponseStatus, EdgeStartTimestamp) are recognised here. Take a slice of a few days rather than a whole month: the 5 MB cap is roughly 25,000 lines at a typical 200 bytes per line.
How do I know a hit is really GPTBot and not someone pretending?
You cannot tell from the user agent string alone; any client can send it. The honest check is the source IP address: Google publishes its crawler IP ranges and a reverse-DNS method, and OpenAI publishes IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User on its crawler page. This tool reads strings only and does not look up or resolve addresses, so treat every row as "requests that claimed to be this agent". If a bot you blocked keeps appearing with 200 responses from addresses outside the published ranges, it is probably not the bot it claims to be.
Should I block the bots I see?
That is a trade-off, not a rule. Blocking a training crawler keeps your text out of future model training by that operator but has no effect on models already trained, and blocking a search-index crawler removes you from that assistant’s search results. User-triggered fetchers answer a person’s question about your page, so blocking them stops that visit. Decide per agent against your own goals, then check the result with the AI crawler access checker, which reads your robots.txt and fetches as each agent. This tool only shows who came and what they received.
Last reviewed