Crawling, rendering, indexation, architecture, migrations and Core Web Vitals.
What an access log line contains, how to match the user agent tokens OpenAI, Anthropic, Perplexity, Google, Bing, Apple and Common Crawl document, how to confirm a request is genuine with published IP ranges or reverse DNS, what to measure, and what logs cannot tell you.
What the llms.txt proposal defines, what Google says about it, what the AI crawler documentation from OpenAI, Anthropic and Perplexity says (nothing), and how to decide whether to publish one.
What Shopify and WordPress with WooCommerce generate for you (robots.txt, sitemaps, canonicals, structured data, robots meta), what you can change, and a checklist for Google and AI crawlers, from the platforms’ own documentation.
Googlebot renders JavaScript in a second pass. The documented crawlers from OpenAI, Anthropic and Perplexity describe no rendering step. What that means for sites built on client-side frameworks, how to check your own pages, and what to serve in the HTML.
A working checklist for a technical SEO audit that covers AI crawlers as well as Googlebot: access, rendering, indexation, architecture, structured data and measurement, with the documented rule behind each check and the free tool that runs it.
The AI crawlers from OpenAI, Anthropic, Perplexity and Google do different jobs. Here is how to separate training, search and user fetches in robots.txt, with a decision table you can copy.