Google-Extended
A robots.txt token, not a crawler. It decides whether content Google crawls may train Gemini and ground Gemini Apps and Vertex AI. It does not affect Search.
| Operator | |
|---|---|
| Type | AI, robots.txt token only: AI training |
| robots.txt token | Google-Extended |
| Follows robots.txt | Yes, by the operator's account; Crawl-delay not honoured |
| Sends visitors | No |
| Links to your pages | No |
| Can end up in AI training | Yes |
| Verification |
None published
Our tracker marks every request as fake, because no real request uses this name. |
| Our call |
Publisher: Block E-commerce: Allow Expert site: Allow What the profiles mean |
How to recognize it
You will not see it in logs. Google says Google-Extended has no user agent string of its own; the crawling is done by Google’s existing crawlers, mostly Googlebot. Any request whose user agent contains Google-Extended is fake.
- robots.txt token:
Google-Extended - IP list: none, because no request carries the name.
- Reverse DNS: none, for the same reason.
What it does
Google-Extended is a robots.txt token that lets a site decide whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. A Disallow for it does not change what Googlebot crawls or what appears in Google Search.
What you gain by allowing it
- Your content in what future Gemini models know, and as a source when Gemini Apps and Vertex AI ground their answers on the web.
What you lose
By allowing it
- Content used for Gemini training with no traffic or payment in return.
By blocking it
- Nothing in Google Search. Google says the token does not affect inclusion or ranking in Search.
- Grounding in Gemini Apps and Vertex AI. That is a channel where answers can point to your pages, so blocking costs an expert site more than it costs a publisher.
- It does not take you out of AI Overviews or AI Mode, which use the Search index. Their only controls are the Search ones (
nosnippet,data-nosnippet,max-snippet,noindex).
How to verify it
There is nothing to verify: no real request uses this name. Our trackers mark every such request as a fake on arrival.
Gotchas and history
- AI Overviews: blocking Google-Extended to stay out of AI Overviews does not work. That is the most common mix-up with this token.
- It is not a bot: a firewall rule matching the
Google-Extendeduser agent blocks only fakes. The real decision is in robots.txt.
What we observed
Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).
- No request carried the name in those 8 days; even the Google Cloud scanner skipped it on these sites.
- On our test bed, 6 requests did between May and August, all fakes from one impersonation pool; see One IP claimed to be Google, OpenAI, Anthropic, Apple and Meta.
About Google
- Common crawlers (Googlebot and its Image and Video variants, GoogleOther, Storebot-Google and others) crawl automatically for Google products and always obey robots.txt. Reverse DNS:
crawl-*.googlebot.comorgeo-crawl-*.geo.googlebot.com. - Special-case crawlers (AdsBot, Mediapartners and others) work for products where the site has an agreement with Google. AdsBot, for example, ignores the
*group in robots.txt with the advertiser’s permission. Reverse DNS:rate-limited-proxy-*.google.com. - User-triggered fetchers fetch a page because a user asked a Google tool or product to. Reverse DNS:
*.gae.googleusercontent.comorgoogle-proxy-*.google.com. - Google-Extended is a robots.txt token, not a crawler. It decides whether content fetched by the crawlers above may train Gemini models and ground Gemini Apps and Vertex AI. It does not affect inclusion or ranking in Google Search.
Each group has its own reverse DNS pattern and IP list. A verifier that accepts every Google domain for every Google user agent lets Google Cloud VMs through (see the Googlebot card).