| Operator | |
|---|---|
| Purpose | robots.txt token controlling use for Gemini training and grounding |
| robots.txt token | Google-Extended |
| Respects robots.txt | It is itself a robots.txt control. |
| If you block it | Opts out of Gemini model training. Google states it does not affect inclusion or ranking in Google Search. |
robots.txt examples
Block Google-Extended
User-agent: Google-Extended
Disallow: /Allow Google-Extended, block a folder
User-agent: Google-Extended
Disallow: /private/
Allow: /A group for a specific crawler replaces the User-agent: * group for that crawler — rules are not combined. Repeat any rules from * that should still apply.
How to verify Google-Extended
Not applicable — no separate requests.
The user-agent alone proves nothing — it is the first thing scrapers copy. See how to verify crawlers.
Is Google-Extended blocked on your site?
Your robots.txt may allow Google-Extended while your CDN or firewall refuses it. Test the real response with the AI crawler checker.
Official documentation: developers.google.com