JavaScript rendering and AI crawlers: what they can and cannot see

The short answer. Google documents a two-phase process in which Googlebot fetches a page’s HTML first and renders its JavaScript later, so content that only appears after scripts run is indexed, but late. The crawler documentation published by OpenAI, Anthropic and Perplexity describes no rendering step at all. If your page’s main text, prices, links or structured data exist only after JavaScript runs, assume those crawlers do not see them, and check it rather than hope.

This article explains what each vendor documents, how to test your own pages in two minutes, and what to change if the test fails.

What does Google document about rendering?

Google’s JavaScript SEO basics describes three phases: crawling, rendering and indexing. Googlebot fetches the HTML, then queues the page for rendering in a headless Chromium, and only after rendering does it see content that JavaScript added. Google notes that rendering is deferred until resources allow, and that pages which rely on JavaScript for their primary content are indexed once rendering completes.

Three practical consequences follow from Google’s own page:

1. Content added by JavaScript can be indexed, later. For most sites the delay is acceptable; for news or fast-changing pages it may not be.
2. Links must be <a href> elements. Google’s crawlable links guidance states that it does not follow links created with onclick handlers, buttons or other non-anchor elements.
3. Errors stop the render. A script that throws before the content mounts leaves Googlebot with the empty shell.

What do the AI vendors document about rendering?

Nothing. That is the finding, and it is worth stating precisely.

• OpenAI’s crawler documentation describes GPTBot, OAI-SearchBot and ChatGPT-User, their purposes, user-agent strings and robots.txt behaviour. It does not describe executing JavaScript.
• Anthropic’s crawler documentation describes ClaudeBot, Claude-SearchBot and Claude-User in the same terms. No rendering step is mentioned.
• Perplexity’s crawler documentation describes PerplexityBot and Perplexity-User. No rendering step is mentioned.

Absence from documentation is not proof of absence in practice, and vendors change their systems without notice. But it is the only documented position, and it matches what we see when we compare the HTML a page serves with what a user-fetch agent is able to quote. The safe assumption is that these agents read the HTML response and nothing more.

• Googlebot

  • Crawl: fetch the HTML
  • Render: run the JavaScript in headless Chromium, later
  • Index: content added by scripts is seen after rendering

  Content added by JavaScript is indexed, but late

• OpenAI, Anthropic and Perplexity crawlers

  • Purposes, user agents and robots.txt behaviour are documented
  • No rendering step is described
  • Safe assumption: they read the HTML response only

  Text, links or JSON-LD added by scripts may never be seen

Why does this matter more than it used to?

Because the retrieval step now feeds an answer, not a list. When Googlebot indexes your JavaScript-rendered content a few days late, the page still ends up in the index. When an assistant’s crawler fetches the HTML shell of a client-rendered page, it finds a title, perhaps a meta description, and a <div id="root">. There is nothing to quote, so the assistant quotes a competitor whose page had the text in the HTML. The user never sees that your page existed.

The same applies to structured data. Google’s structured data guidelines accept JSON-LD injected by JavaScript because Google renders; a crawler that does not render never sees it. If your Organization or Article markup is added client-side, the AI crawlers get a page with no declared entity.

How do you check your own pages?

Two minutes per template.

1. Fetch the raw HTML. curl -A "Mozilla/5.0" https://example.com/page/ or view source in the browser (not the inspector, which shows the rendered DOM). Search it for a sentence from the main content, a price, the first internal link in the body, and application/ld+json.
2. Compare with JavaScript disabled. Turn scripts off in the browser and reload. What disappears is what a non-rendering crawler never had. Our no-JS content check does this comparison for one URL and lists the headings, text, links and JSON-LD that vanish.
3. Fetch as the agents. Request the page with each documented user-agent string and confirm the response is 200 and contains the same HTML; a CDN rule can serve something different to bots. Our AI crawler access tool does this for the documented agents.
4. Check Google’s view separately. The URL Inspection tool in Search Console shows the rendered HTML Googlebot produced, which is the right test for Google and the wrong test for everyone else.

If step 2 removes the main content, you have a rendering dependency. If step 3 shows a different response per agent, you have an access problem in front of it; fix that first.

What should you serve in the HTML?

Everything a crawler needs to understand and quote the page:

• The main text, headings and the first screen of content.
• Internal links as real anchors with href attributes.
• Title, meta description, canonical and robots directives.
• JSON-LD structured data.
• Prices, dates, author names and any fact you would want quoted correctly.

JavaScript can still do everything else: interactivity, personalisation, analytics, lazy sections below the fold. The principle is that scripts enhance a page that already works; they should not be the page.

How do you fix a rendering dependency?

In rough order of cost:

1. Static generation for pages whose content changes rarely: marketing pages, documentation, articles. The framework renders HTML at build time; the server serves files.
2. Server-side rendering for pages that change per request or per user. The server renders HTML for each request; the client takes over afterwards.
3. Hybrid rendering: the frameworks most teams already use (Next.js, Nuxt, SvelteKit, Astro, Remix among others) support static, server and client rendering per route, so the fix is usually configuration and a review of which routes need which mode, not a rewrite.
4. Pre-rendering or dynamic rendering, serving a rendered snapshot to bots, is the least attractive option. It creates two versions of the site to keep in sync and depends on detecting bots correctly, which the user-agent and IP lists make possible but fragile.

Whichever you choose, re-run the check in the previous section on every template after the change, and add it to your release checklist so the next deployment does not quietly reintroduce the dependency.

What we don’t claim

• We do not claim to know what any vendor’s crawler does beyond what it documents. The statements above are drawn from the linked documentation, which can change without notice.
• We do not claim that serving content in HTML causes an assistant to cite it. It makes citation possible; whether it happens depends on the content and on the engine.
• We do not claim that client-side rendering is wrong for every application. It is wrong for pages you want retrieved and quoted.