How to recognize it

User agents, as they reach our logs:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

In 8 days it used 11 different browser prefixes: Chrome, Edge and Firefox on Windows, Mac and Linux. Only the (compatible; meta-externalagent/1.1 ...) tail stays the same, so match on meta-externalagent. Meta’s docs show the bare form, meta-externalagent/1.1, with a relative link.

  • robots.txt token: meta-externalagent
  • IP list: none published. Meta’s crawler page says IP allowlisting is more secure but points to its documentation index or [email protected] for the addresses. We check Meta’s own network, AS32934.
  • Reverse DNS: none published.

What it does

Meta says this crawler collects content for uses such as training its foundation AI models. To block it, Meta tells site owners to disallow it in robots.txt.

What you gain by allowing it

  • Presence in what Meta’s models learn from. There is no link or click.

What you lose

By allowing it

  • Content used for training with no traffic or payment in return.
  • Load, heavy on images: on our WordPress site 77% of its requests were for image files.

By blocking it

  • Nothing in Meta AI search or in link previews, which use other user agents.
  • Presence in Meta’s training data. A publisher that licenses content keeps leverage; an expert site gives up a channel.

How to verify it

Check that the IP belongs to Meta’s network, AS32934. That proves the request came from Meta, not which Meta service sent it. Cloudflare flags the real crawler as a verified bot in the category “AI Crawler”, and in our data the flag matched the ASN check on every request.

Gotchas and history

  • Shared-hosting firewalls: a firewall can throttle it hard below the CDN. On our WordPress site the host answered 44% of its requests with 429 (see below).
  • JavaScript: on our test bed it ran JavaScript on most page fetches; see Meta-ExternalAgent ran our JavaScript. On the WordPress site below it fetched no scripts or stylesheets at all.
  • No robots.txt reads: it did not request robots.txt once in 8 days here, nor on our test bed in May; see Most AI batch crawlers never fetched our robots.txt.

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 1,403 requests carried a Meta-ExternalAgent user agent:
    • 680 (48%) came from Meta’s network (AS32934);
    • 717 of the 723 others came from the Google Cloud scanner;
    • Cloudflare’s flag matched the ASN check on all 1,403.
  • All 680 Meta requests went to the WordPress site:
    • 523 (77%) were for images and none for scripts or stylesheets;
    • 300 (44%) got a 429 from the hosting firewall;
    • none asked for /robots.txt.

About Meta

  • meta-externalagent crawls for uses such as training Meta’s foundation AI models.
  • meta-webindexer crawls to improve Meta AI’s search results, and allowing it helps Meta AI cite and link your content.
  • meta-externalfetcher fetches individual links at a user’s request, and may bypass robots.txt.
  • facebookexternalhit builds link previews when a page is shared, and may bypass robots.txt for security or integrity checks.
  • meta-externalads crawls for Meta’s advertising and business products. We have not seen it in our logs.

The training and search crawlers have separate tokens. To be cited by Meta AI without feeding training, allow meta-webindexer and disallow meta-externalagent.

Meta publishes no IP list or reverse DNS scheme on its crawler page. It calls IP allowlisting the more secure option, but refers site owners to [email protected] for the addresses. A check against Meta’s network, AS32934, proves a request came from Meta, but not which of these agents sent it.

Sources