What is Google-Extended?
Google-Extended is a robots.txt product token that lets site owners control whether content Google crawls from their site may be used to train future Gemini models and to ground Gemini answers. Google introduced it in 2023. It does not affect whether or how your pages appear in Google Search.
Unlike most AI crawlers, Google-Extended is not a separate bot with its own user agent. Google keeps crawling with its existing crawlers, and the token only tells it how the content may be used. Grounding here means giving a model content from Google's index at the moment it answers, to make responses more factual and relevant. According to Google's documentation, the token covers Gemini Apps and the Gemini API on Vertex AI, including Grounding with Google Search.
Why Google-Extended matters for AI visibility
A common misunderstanding is that blocking Google-Extended removes you from Google's AI answers. It does not. Google has said the token does not affect inclusion or ranking in Google Search, and AI features in Search, such as Google AI Overviews, are part of Search. To limit those, you would need regular Search controls such as nosnippet, which also affect your normal listings.
The trade-off lies in Gemini itself. Blocking Google-Extended keeps your content out of future Gemini training and out of grounding in Gemini Apps, so Gemini may draw on your pages less often when people ask about your category. For brands that want to be visible in Gemini, that is usually a cost rather than a benefit.
How to use Google-Extended
Decide what you want: allow or block the use of your content for Gemini training and grounding. The token covers both at once.
To opt out, add a group to your robots.txt with "User-agent: Google-Extended" and "Disallow: /", or disallow only specific folders.
Leave Googlebot alone: blocking it would stop Google from crawling your pages for Search.
Do not look for the token in your server logs: Google does not crawl under that name.
Track how Gemini describes your brand before and after the change.
Trackbase tracks your visibility in Gemini alongside Google AI Overviews and Google AI Mode, so you can see whether a robots.txt change affects how Gemini presents your brand.
Examples of Google-Extended
A news publisher blocks Google-Extended to keep its archive out of Gemini training, knowing its articles can still appear in Google Search and AI Overviews.
A B2B software vendor finds that a copied robots.txt template blocked Google-Extended, and removes the rule because it wants Gemini to use its documentation.
A Dutch bicycle webshop keeps Google-Extended allowed, so Gemini can use its buying guides when people ask which e-bike to buy.
A stock photo site blocks Google-Extended only for the folder with its licensed image descriptions and leaves the rest of the site open.
Frequently asked questions
Does blocking Google-Extended remove my site from AI Overviews?
No. Google has said Google-Extended does not affect inclusion or ranking in Google Search, and AI Overviews are part of Search. Blocking the token only limits the use of your content for training and grounding Gemini models. To limit what appears in AI features in Search, you would need Search controls such as nosnippet, which also affect regular results.
Is Google-Extended a crawler?
No. It is a product token you use in robots.txt, not a separate bot. Google crawls with its existing user agents, such as Googlebot, and reads the Google-Extended rules to decide how the crawled content may be used. That is also why you will not find Google-Extended in your server logs.
Should I block Google-Extended?
Only if keeping your content out of Gemini training and grounding matters more to you than being used by Gemini. Some publishers block it to protect their archives. For brands that want to appear in AI answers, allowing it is usually the better choice, since blocking it does not keep them out of AI Overviews and may reduce how often Gemini draws on their pages.