AI search visibility

How Gemini chooses sources: what Google documents, and what it does not

What Google publishes about the sources behind answers in the Gemini app: grounding in Google Search, the Google-Extended token and what it does not control, the Sources button, and the questions its documentation leaves open. With a checklist.

The short answer. Google says “some Gemini responses are grounded on Search results”, and that Gemini “can always use public information from Google Search”. So the route into a Gemini app answer runs through the Google Search index, which Googlebot builds. The one Gemini-specific control is the Google-Extended robots.txt token, which Google says covers grounding in Gemini Apps as well as training, and which “does not impact a site’s inclusion in Google Search”. Google does not document how the Gemini app chooses which sources to use or link, and there is no Gemini report in Search Console.

This article is about the Gemini app at gemini.google.com and its mobile apps. For AI Overviews and AI Mode inside Google Search, which have their own guidance, see what Google says matters for AI Overviews and AI Mode. Everything below comes from Google’s common crawlers page, five Gemini Apps Help articles and the Gemini API’s Grounding with Google Search page, as read on 15 October 2026.

Where does the Gemini app get web content from?

From Google Search, as far as Google’s help pages say. Three of them state it in slightly different ways:

  • The Gemini Apps Privacy Hub: “We rely on other Google services and infrastructure to provide Gemini Apps. For example, some Gemini responses are grounded on Search results.”
  • The Connected Apps page lists Google Search first among the services Gemini “automatically uses public information from”, and adds: “Gemini can always use public information from Google Search, even if Keep Activity is off or you’re signed out.”
  • The Deep Research page: “By default, Gemini includes Google Search as a source for your research.” A user can deselect it and research only their own Gmail, Drive or uploaded files.

“Some” is the important word. Google does not say which responses are grounded, or when Gemini decides a search is needed.

Which Google crawler matters for Gemini?

Googlebot, plus a token that is not a crawler. Google’s common crawlers page describes both:

Name What Google says What it changes
Googlebot Crawling preferences addressed to it “affect Google Search (including Discover and all Google Search features)” and other products such as Google Images and Google News Blocking it keeps the page out of the Search index, which is where Gemini’s grounding content comes from
Google-Extended “A standalone product token” for managing whether crawled content is used for training Gemini models and “for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps” Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”

Two details on that page matter in practice. Google-Extended “doesn’t have a separate HTTP request user agent string”: crawling is done by Google’s existing crawlers, and the token is read from robots.txt only “in a control capacity”. So you will never see Google-Extended in your server logs, and you cannot block it at the firewall. And Googlebot is the access question: the grounding Google describes uses “the Google Search index”, so a page Googlebot cannot crawl has no route in. That last step is our reading of the two statements together; Google does not spell it out for the Gemini app.

Our reference pages for Googlebot and Google-Extended keep the details in one place.

What does Google-Extended control, and what does it not?

It controls more than training. People often treat Google-Extended as a training opt-out only. Google’s own description covers two uses: training “future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini”, and grounding “in Gemini Apps and Grounding with Google Search on Vertex AI”.

So the decision is not as simple as it is for OpenAI’s GPTBot, where training and search sit on separate agents. Disallowing Google-Extended opts out of training and also, on Google’s description, out of your content being provided to Gemini Apps as grounding. Allowing it opts in to both. Google documents no setting that separates them.

What it does not control, per the same page and the AI features page:

  • Google Search. Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
  • AI features inside Search. The AI features page says “robots.txt directives for Googlebot is the control” for Search, and points to Google-Extended only “to limit AI training and grounding in some of Google’s other systems”. AI Overviews and AI Mode follow Googlebot and the snippet controls, covered in our article on them.

What Google does not say: whether a page from a site that disallows Google-Extended can still be shown as a link in a Gemini app answer, for example in the related links. The page describes the content supplied to the model, not the links shown to the user.

  1. Crawled

    Googlebot fetches the page.

    Stopped by: a Disallow for Googlebot, or a CDN or WAF block

  2. In the Search index

    The same index that serves Google Search.

    Stopped by: noindex

  3. Grounding content

    Supplied to the model at prompt time when Gemini searches.

    Stopped by: a Disallow for Google-Extended

  4. Linked as a source

    Shown under the Sources button or in-line, for some responses.

    Not documented: how sources are chosen

Separately: Search

Google-Extended does not affect inclusion or ranking in Google Search, including AI Overviews and AI Mode.

The path from your page to a Gemini app answer, as far as Google documents it. How sources are chosen and linked is not documented.

Sometimes. The help article View related sources from Gemini Apps says Gemini “may provide links to sources and related content for you to dig deeper”, and that these can include “public websites”, uploaded files, or Workspace documents and emails. When sources exist, a Sources button appears “at the bottom of the response or in-line throughout the response” and opens a side panel.

The same page sets three limits worth knowing:

  • “Not all responses include related links or sources.” No Sources button means Gemini provided no links for that response.
  • “If Gemini Apps directly quotes a large amount of text from a webpage, you’ll see a link to that webpage in the sources list.” That is the one case where the page says a link will appear.
  • If a response includes an image from the web, Gemini “will show the source and provide a link to it”.

The page calls these “sources and related content”, and says the links “may include content that is related to parts of Gemini’s response”. It does not say a listed link is the page the answer was built from. If you have read about a “double-check response” feature, note that the version of this help page we read does not mention one, so we do not describe it here.

One more caution from Learn about responses from Gemini Apps: Google says Gemini “can misrepresent how it works”, including when asked how it carries out functions “like citing sources, or providing fresh information”. Asking Gemini why it chose a source is not evidence of how it chose it.

What does the Gemini API documentation add?

A description of the grounding mechanism, with a caveat: the Gemini API is a developer product, not the Gemini app, and Google does not say the app works the same way.

The Grounding with Google Search page says the feature “connects the Gemini model to real-time web content”, and sets out the steps when a developer enables the google_search tool:

  1. The model “analyzes the prompt and determines if a Google Search can improve the answer.”
  2. If needed, it “automatically generates one or multiple search queries and executes them.”
  3. It “processes the search results, synthesizes the information, and formulates a response.”
  4. The response comes back with inline citations: each citation “links a text segment … to a source URL”.

That is useful as a picture of how grounding in Search can work: the model writes its own search queries, so a page can be found for a sub-question the user never typed. The API page also gives developers “complete control over how you display sources”, which is a reminder that how many links appear is a product decision, separate from which pages were retrieved.

Two gaps follow. The API page does not describe the Gemini app. And Google’s crawlers page names “Grounding with Google Search on Vertex AI” as covered by Google-Extended; it does not name the Gemini API on ai.google.dev, so we cannot tell you from the documentation whether the token applies there.

What does Google not document?

  • Selection and ranking. None of the pages linked here explains how the Gemini app decides when to search, which results it uses, or which it shows as links.
  • A Gemini report. Search Console’s Generative AI performance report covers AI Overviews and AI Mode in Google Search. It does not cover the Gemini app, and Google offers site owners no other report of Gemini appearances.
  • Special markup or files. Nothing in these pages asks for structured data, llms.txt or a Gemini-specific sitemap.
  • How fast a change to Google-Extended takes effect. The crawlers page gives no timing.

Measure it from the outside instead. Visits arrive with gemini.google.com as the referrer, and our AI referral segment tool groups them with other assistants in GA4. For appearances, mentions and accuracy, add Gemini to a sampled prompt set run on a fixed schedule; our Lab uses the same method.

In our case studies the visit volume is very small. On shopifywebdeveloper.co.uk, GA4 recorded chatgpt.com “16 times in 201 sessions (8%) and Gemini once”. That is one session, too few to say anything about Gemini as a channel; we report it so the figure is not mistaken for something larger.

A checklist

  1. robots.txt allows Googlebot on the pages you want used (check the live file).
  2. Your CDN, WAF and bot protection let verified Googlebot through. Our AI crawler access tool fetches a URL as Googlebot and reports what it receives.
  3. Pages you want used are indexed, with no noindex.
  4. You have decided on Google-Extended knowing that, on Google’s description, it covers grounding in Gemini Apps as well as training. It cannot be handled at the firewall, only in robots.txt.
  5. Pages answer specific questions directly in HTML text, since the model writes its own search queries.
  6. Analytics groups gemini.google.com referrals into a report you check monthly.
  7. Gemini is in your sampled prompt set, scored for mentions, citations and accuracy.
  8. Calendar a quarterly re-read of the common crawlers page and the Gemini Apps Help articles linked here.

What we don’t claim

  • We do not claim to know how the Gemini app selects or links sources. Google does not document it in the pages linked here, and we have marked where our reading begins.
  • We do not claim the Gemini API’s grounding steps describe the Gemini app. Google documents them for developers only.
  • We do not claim that any step above causes a mention or a link. They keep a page eligible to be used.
  • We do not claim the details are permanent. Vendor documentation changes without notice; re-read the sources before relying on them.

Sources

  1. Google Search Central — Google's common crawlers
  2. Gemini Apps Help — View related sources from Gemini Apps
  3. Gemini Apps Help — Use & manage Connected Apps in Gemini
  4. Gemini Apps Help — Use Deep Research in Gemini Apps
  5. Gemini Apps Help — Gemini Apps Privacy Hub
  6. Gemini Apps Help — Learn about responses from Gemini Apps
  7. Gemini API — Grounding with Google Search
  8. Google Search Central — AI features and your website
AI search visibility

How Claude chooses sources: what Anthropic documents, and what it does not

What Anthropic publishes about its three crawlers, robots.txt, its IP list, web search in the Claude apps and the web search tool in the Claude API, what it leaves open about how sources are chosen, and a checklist for being eligible as a cited source.

7 min read