How to recognize it

User agents, as they reach our logs:

AdsBot-Google (+http://www.google.com/adsbot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
  • First string: the desktop crawler.
  • Second string: the mobile one. Google’s page lists it both as active and among retired crawlers, and it made almost half the requests on our sites.

Match on AdsBot-Google, which also catches the mobile variant.

  • robots.txt tokens: AdsBot-Google and AdsBot-Google-Mobile
  • IP list (JSON): special-crawlers.json
  • Reverse DNS: rate-limited-proxy-*.google.com

What it does

AdsBot checks the quality of web pages for Google Ads. Google lists it among special-case crawlers and says it ignores the global * group in robots.txt with the advertiser’s permission. Rules for it apply only in a group that names it.

What you gain by allowing it

  • Google Ads can check your landing pages, which affects whether and how your ads run. On a site that does not advertise with Google, it has little reason to visit.

What you lose

By allowing it

  • Little load: 109 requests in 8 days on our two sites.

By blocking it

  • Google Ads loses its view of your landing pages, which can hurt ads that point to them. Blocking needs a group naming AdsBot-Google, and a separate one for AdsBot-Google-Mobile.

How to verify it

Forward-confirmed reverse DNS, as for Googlebot, or a match against special-crawlers.json. Cloudflare flags the real crawler as a verified bot in the category “Advertising & Marketing”, and in our data the flag matched our check on every request.

Gotchas and history

  • The * group: User-agent: * with Disallow: / does not stop it, because AdsBot ignores that group.
  • Two tokens: a group for AdsBot-Google does not cover AdsBot-Google-Mobile, which has its own token. Google also lists the mobile crawler as retired, but it is still active.
  • Not a scanner costume: unlike most crawler names, no fake AdsBot reached our sites.

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 109 requests, every one verified by forward-confirmed reverse DNS (rate-limited-proxy-*.google.com) and by Cloudflare’s flag. No fakes.
  • 61 used the desktop string and 48 the mobile one.
  • The visits came on only 3 of the 8 days, in short bursts across a few dozen pages.

About Google

  • Common crawlers (Googlebot and its Image and Video variants, GoogleOther, Storebot-Google and others) crawl automatically for Google products and always obey robots.txt. Reverse DNS: crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com.
  • Special-case crawlers (AdsBot, Mediapartners and others) work for products where the site has an agreement with Google. AdsBot, for example, ignores the * group in robots.txt with the advertiser’s permission. Reverse DNS: rate-limited-proxy-*.google.com.
  • User-triggered fetchers fetch a page because a user asked a Google tool or product to. Reverse DNS: *.gae.googleusercontent.com or google-proxy-*.google.com.
  • Google-Extended is a robots.txt token, not a crawler. It decides whether content fetched by the crawlers above may train Gemini models and ground Gemini Apps and Vertex AI. It does not affect inclusion or ranking in Google Search.

Each group has its own reverse DNS pattern and IP list. A verifier that accepts every Google domain for every Google user agent lets Google Cloud VMs through (see the Googlebot card).

Sources