How to recognize it

User agents, as they reach our logs:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • First string: the smartphone crawler, which fetches the pages. The Chrome version varies; we saw four different ones in 8 days.
  • Second, short string: it made every robots.txt request in our data.
  • Desktop crawler: documented by Google, but absent from our logs in those 8 days:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Match on the token Googlebot, then verify the IP. The user agent alone proves nothing.

  • robots.txt token: Googlebot
  • IP list (JSON): common-crawlers.json
  • Reverse DNS: crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com

What it does

Googlebot crawls pages for Google Search. Google documents that rules for the Googlebot token apply to Search, including Discover and all Search features, and also to Google Images, Video, News and Discover.

The AI features in Search (AI Overviews, AI Mode) have no crawler of their own. To be a supporting link in them, a page must be indexed and eligible to show with a snippet in Search, which means Googlebot has to be able to fetch it.

What you gain by allowing it

  • Organic traffic from Google Search, Discover, Images and News.
  • Supporting links in AI Overviews and AI Mode. Google counts clicks from them in the overall Search traffic in Search Console.

What you lose

By allowing it

  • Crawl load. On our sites it made 2,502 verified requests in 8 days, second only to ClaudeBot.
  • Gemini use. Content that Googlebot fetches can be used to train Gemini models and for grounding in Gemini Apps and Vertex AI, unless you disallow the separate Google-Extended token. That token does not affect Search.
  • No AI-only opt-out. Google offers no token that removes a page from AI Overviews or AI Mode alone. The documented controls are the Search ones (nosnippet, data-nosnippet, max-snippet, noindex), and they apply to regular results as well.

By blocking it

  • Google Search traffic, including Discover and the AI features.
  • Blocking does not reliably remove pages. A URL disallowed in robots.txt can still be indexed if other sites link to it, and shows up without a description. Google’s advice for keeping a page out is noindex or a password, and noindex only works when the page can be crawled.

How to verify it

Forward-confirmed reverse DNS: the hostname must end in googlebot.com (crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com), and resolving that hostname must return the original IP.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

Or match the IP against common-crawlers.json. Cloudflare’s verified-bot flag (category “Search Engine Crawler”) agreed with forward-confirmed reverse DNS on every request in our data.

Do not accept googleusercontent.com for Googlebot. Google’s generic verification steps list it next to googlebot.com and google.com, but it belongs to user-triggered fetchers (*.gae.googleusercontent.com). Any Google Cloud VM gets a reverse DNS name under bc.googleusercontent.com that passes the forward check (see What we observed).

Gotchas and history

  • Crawl-delay is not supported.
  • Google caches robots.txt for up to 24 hours, longer if it cannot refresh it.
  • How robots.txt errors are read:
    • Any 4xx except 429 is read as “no robots.txt”, so a firewall that answers robots.txt with 403 opens the whole site.
    • A 5xx stops crawling for the first 12 hours.
  • Country rules: Google crawls mostly from US addresses, but may switch to other countries when US requests are blocked. Exempt verified bots (for example Cloudflare’s cf.client.bot) instead of allowing countries.
  • Classifier pitfall: the string Googlebot also appears in other user agents, such as Googlebot-Image, Googlebot-Video and a media-monitoring bot that announces itself as “compatible with Googlebot”. Match the specific ones first.
  • The crawler docs moved from developers.google.com/search/docs to developers.google.com/crawling. The older IP list at /static/search/apis/ipranges/googlebot.json still answered on 2026-10-26, but the docs now point to common-crawlers.json.

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 4,051 requests carried a Googlebot user agent. Of those:
    • 2,502 (62%) passed forward-confirmed reverse DNS and 1,549 (38%) failed;
    • Cloudflare’s verified-bot flag gave the same verdict as ours on all 4,051.
  • 1,446 of the 1,549 failures came from Google Cloud (AS396982), from 92 IP addresses with reverse DNS names under bc.googleusercontent.com. They are the scanner that also poses as 27 other crawlers.
    • In a sample of 5, the forward lookup returned the original IP.
    • A check that accepts all three domains from Google’s generic steps for every Google user agent would have passed each sampled request as Googlebot.
    • The most requested paths were /, /.env, /@fs/.env and /api: vulnerability scanning behind a Googlebot user agent.
  • Verified Googlebot requested robots.txt 483 times across 4 hostnames, about 15 times per hostname per day, despite the 24-hour cache. All 483 used the short user agent, which made only 117 other requests. The 1,902 page requests used the smartphone string, and the desktop crawler did not appear.

About Google

  • Common crawlers (Googlebot and its Image and Video variants, GoogleOther, Storebot-Google and others) crawl automatically for Google products and always obey robots.txt. Reverse DNS: crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com.
  • Special-case crawlers (AdsBot, Mediapartners and others) work for products where the site has an agreement with Google. AdsBot, for example, ignores the * group in robots.txt with the advertiser’s permission. Reverse DNS: rate-limited-proxy-*.google.com.
  • User-triggered fetchers fetch a page because a user asked a Google tool or product to. Reverse DNS: *.gae.googleusercontent.com or google-proxy-*.google.com.
  • Google-Extended is a robots.txt token, not a crawler. It decides whether content fetched by the crawlers above may train Gemini models and ground Gemini Apps and Vertex AI. It does not affect inclusion or ranking in Google Search.

Each group has its own reverse DNS pattern and IP list. A verifier that accepts every Google domain for every Google user agent lets Google Cloud VMs through (see the Googlebot card).

Sources