Technical SEO audit checklist for Google and AI crawlers

The short answer. A technical audit in 2026 has to answer one question for two audiences: can Googlebot, and can the documented AI crawlers, reach, read and understand every page that matters? The checks below are in dependency order. Do not skip ahead: a rendering problem makes every content finding moot, and an access problem makes the rendering check moot.

Each item names the documented rule it tests and, where we have one, the free tool that runs the check. Use it as a working document; tick items as you verify them, not as you assume them.

1. 1

   Access

   Can crawlers fetch the pages?

2. 2

   Rendering

   Can crawlers read the content?

3. 3

   Indexation

   Are the right pages in the index?

4. 4

   Architecture

   Can crawlers find everything?

5. 5

   Understanding

   Do engines know what the pages and the company are?

6. 6

   Measurement

   Can you tell whether any of this worked?

Stage 1: Access. Can crawlers fetch the pages?

• robots.txt parses and says what you think it says. Google documents how it interprets the file, including group precedence and the longest-match rule for Allow and Disallow (robots.txt specification). Read the live file, not the one in the repository.
• Each documented AI agent is treated as intended. OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User) and Perplexity (PerplexityBot, Perplexity-User) each document separate agents for training, search indexing and user fetches. A rule written for one does not apply to the others. Tool: AI crawler access.
• The CDN or bot protection does not override robots.txt. Fetch key URLs as each agent and compare the status code with what robots.txt allows. A 403 from the edge is invisible in the file.
• Key pages return 200, once. No redirect chains, no soft 404s, no 5xx under load. Google’s redirects documentation covers which redirect types pass signals. Tool: redirect chain checker.
• Logs show the crawlers actually arriving. Verify agents against published IP ranges; Google documents how to verify Googlebot, and the AI vendors publish their ranges. Tool: AI bot log analyser.

Stage 2: Rendering. Can crawlers read the content?

• The main content is in the HTML response. Google renders JavaScript in a second pass (JavaScript SEO basics); the AI vendors’ crawler documentation describes no rendering step at all. Fetch each template with JavaScript disabled and compare. Tool: no-JS content check. We go deeper in JavaScript rendering and AI crawlers.
• Links are real anchors with href. Google’s crawlable links guidance is explicit: buttons and JavaScript handlers are not followed. Tool: page links audit.
• Structured data is in the server HTML, not injected after load.
• Core Web Vitals field data is “good” at the 75th percentile for the templates that matter, per Google’s thresholds. Lab scores are for diagnosis only. Tool: Core Web Vitals.

Stage 3: Indexation. Are the right pages in the index?

• Every page that should rank is indexable: no stray noindex, no X-Robots-Tag header left over from staging (robots meta tag documentation). Tool: indexability inspector.
• Every page that should not rank is excluded by noindex, not by robots.txt. A page blocked in robots.txt cannot be seen to carry noindex.
• Canonicals are self-referencing and match the sitemap exactly. Google’s canonicalization documentation treats the link element as a hint, so the variants (http, www, trailing slash, parameters) also need redirects.
• The XML sitemap lists only indexable, canonical, 200 URLs, with real lastmod dates or none (build a sitemap). Tool: sitemap inspector.
• Search Console’s indexing report agrees. Submitted versus indexed, and the reasons for every exclusion.

Stage 4: Architecture. Can crawlers find everything, and does the structure make sense?

• Every important page is within three clicks of the home page and linked from at least one indexable page.
• Faceted navigation, filters and parameters do not create an infinite space. Google’s crawl-budget guidance matters most on large sites, but duplicate URL spaces waste any crawler’s fetches.
• One page per topic. Pages that compete for the same query dilute each other.
• Headings describe the content in order, with one H1 and no skipped levels. Tool: heading outline.
• Titles and descriptions are unique and within length. Tool: title and description preview.

Stage 5: Understanding. Do engines know what the pages and the company are?

• Structured data matches the visible page. Google’s general guidelines require it; markup that disagrees with the text can be ignored or penalised. Tool: structured data check.
• Organization and Person facts agree with the About page, the footer and the profiles. Tool: entity consistency.
• Open Graph tags exist and the image loads. Tool: social preview.
• Author, published and updated dates are visible and consistent with any Article markup. Tool: author and freshness signals.

Stage 6: Measurement. Can you tell whether any of this worked?

• A dated baseline exists: indexed pages, impressions, clicks and Core Web Vitals status from Search Console; the Generative AI performance report; Bing’s AI Performance report where available.
• AI-referral traffic is segmented in analytics. Tool: AI referral segment.
• A sampled AI-visibility baseline exists for the prompts that matter, with the method written down. See the sampling method.
• A second measurement date is in the calendar.

Common findings, in the order they usually matter

1. The CDN blocks an AI agent that robots.txt allows.
2. The main content, or the JSON-LD, only exists after JavaScript runs.
3. Staging noindex or X-Robots-Tag shipped to production on one template.
4. Canonicals point at URLs that redirect.
5. Sitemaps list redirected, noindexed or parameterised URLs.
6. Two or three pages target the same question.
7. Organization facts differ between the About page, the footer and the structured data.
8. No baseline, so no one can say whether last quarter’s work changed anything.

What we don’t claim

• We do not claim a clean checklist produces rankings or citations. It removes the reasons a page cannot be considered; what happens next depends on the content and on the engines.
• We do not claim this list is complete for every platform. E-commerce faceting, international sites and large JavaScript applications each add checks of their own.
• We do not claim any tool replaces reading the logs and the pages. The tools above make the checks faster; they do not make the judgement.