AI search optimization
Be found and described accurately by assistants
- Access
Stage 1: Access
AI crawler access checker
robots.txt verdicts for 16 AI and search agents, then a live fetch as each one to find blocks robots.txt hides.
Free to use. No login, nothing stored.
What it checks
Access- robots.txt verdict per agent, with the matched rule
- The documented purpose of each agent: training, search index, user-triggered fetch or control token
- A live fetch of the homepage as each agent to show firewall or 403 blocks
- Which agents have no user agent to fetch with (control tokens)
Method question for this stage: Can crawlers reach and render every key page?
Run the check
Results
No check run yet
- Info
One row per agent, 16 agents
Each row shows the robots.txt verdict with the matched rule, the agent's documented purpose, and what a live fetch as that agent returned.
Allowing a crawler does not mean you will be cited; it is the first stage of the method.
How we check
The rules this tool tests, and where they come from
A result is a check against a published rule, not a prediction. Every row shows its working.
Rules and sources
- OpenAI: overview of OpenAI crawlers
- Anthropic: does Anthropic crawl data from the web
- Perplexity: crawlers
- Google: overview of Google crawlers and fetchers
Links go to the operator's or standards body's own documentation. We do not link to third-party summaries.
Read more
Next step
Fix what it finds
This check belongs to the Access stage of the Crawl-to-Citation Method. The service below covers the fix; the Diagnostic ranks it against everything else.
Most engagements start with the Search Visibility Diagnostic ($2,500).
It covers all six method stages, so what this tool finds is ranked against everything else before you commit to any work.
Book the diagnosticCrawler access questions
What the verdicts mean, and what they do not.
Still have questions? Book a callDoes allowing a crawler get me cited?
No. Access is necessary, not sufficient. A crawler that can reach your pages still has to understand them, trust the source and find a passage worth quoting. This tool covers the first of those stages only.
Do user-triggered fetchers obey robots.txt?
Operators differ, and their documentation changes. OpenAI, Anthropic and Perplexity each document a user-triggered agent that fetches a page when a person asks for it and that does not crawl on its own; how far each honours robots.txt for those fetches is stated per operator, so every row links to the operator page rather than guessing.
Why check a live fetch too?
robots.txt is a request to the crawler; a firewall or bot-management rule is enforced by your server. A site can allow GPTBot in robots.txt and still answer its requests with a 403 or a challenge page. The live fetch sends each documented user-agent string once and reports what came back.
Last reviewed