How to recognize it

User agents, as they reach our logs:

Mediapartners-Google
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Mediapartners-Google) Chrome/153.0.8010.36 Safari/537.36
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Mediapartners-Google) Chrome/153.0.8010.36 Mobile Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko, Mediapartners-Google) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION]
  • First string: the bare desktop name.
  • Second and third: the desktop and mobile variants. Note that the token sits inside the parentheses after KHTML, like Gecko.
  • Last string: sent with its template placeholders unfilled. Do not treat that as a sign of a fake; the real crawler does it.

Match on the token Mediapartners-Google.

  • robots.txt token: Mediapartners-Google
  • IP list (JSON): special-crawlers.json
  • Reverse DNS: rate-limited-proxy-*.google.com according to Google, but real requests also come from crawl-*.googlebot.com.

What it does

Mediapartners-Google visits pages on sites in Google’s ad programmes, such as AdSense and Ad Manager, to work out which ads fit them. Google lists it among its special-case crawlers, which work for products where the site has an agreement with Google, and says it ignores the global * group in robots.txt.

What you gain by allowing it

  • Ads that match the page, which is what AdSense revenue depends on. On a site without Google ads it has nothing to do.

What you lose

By allowing it

  • A little crawl load. On our AdSense site it made 689 requests in 8 days.

By blocking it

  • Less relevant ads, and less revenue, on a site that runs AdSense or Ad Manager. Blocking it needs a group naming Mediapartners-Google; User-agent: * does not reach it.

How to verify it

Forward-confirmed reverse DNS, as for Googlebot, or a match against special-crawlers.json. Accept both google.com and googlebot.com names: a verifier that accepts only the documented rate-limited-proxy-*.google.com rejects real requests (see below). Cloudflare flags the real crawler as a verified bot in the category “Advertising & Marketing”.

Gotchas and history

  • robots.txt: a Disallow under User-agent: * does not apply. A site that blocks everything with * still gets this crawler.
  • Templates: the unfilled placeholder user agent looks like a broken scraper, but it is Google.
  • No robots.txt reads: in 8 days it did not request robots.txt once on our site.

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 689 requests, all on the one site that shows AdSense ads. Cloudflare flagged all 689 as verified.
  • 680 resolved to rate-limited-proxy-*.google.com. The other 9 resolved to crawl-*.googlebot.com, the names of Google’s common crawlers. Our own check followed the documented pattern and rejected those 9, which Cloudflare accepted, until we widened it on 26 October.
  • 54 requests used the user agent with unfilled [WEBKIT_VERSION] and [CHROME_VERSION] placeholders.
  • None asked for /robots.txt.

About Google

  • Common crawlers (Googlebot and its Image and Video variants, GoogleOther, Storebot-Google and others) crawl automatically for Google products and always obey robots.txt. Reverse DNS: crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com.
  • Special-case crawlers (AdsBot, Mediapartners and others) work for products where the site has an agreement with Google. AdsBot, for example, ignores the * group in robots.txt with the advertiser’s permission. Reverse DNS: rate-limited-proxy-*.google.com.
  • User-triggered fetchers fetch a page because a user asked a Google tool or product to. Reverse DNS: *.gae.googleusercontent.com or google-proxy-*.google.com.
  • Google-Extended is a robots.txt token, not a crawler. It decides whether content fetched by the crawlers above may train Gemini models and ground Gemini Apps and Vertex AI. It does not affect inclusion or ranking in Google Search.

Each group has its own reverse DNS pattern and IP list. A verifier that accepts every Google domain for every Google user agent lets Google Cloud VMs through (see the Googlebot card).

Sources