Meta-WebIndexer
Meta's crawler for Meta AI search results. Meta says allowing it helps Meta AI cite and link your content.
| Operator | Meta |
|---|---|
| Type | Search engine: AI search |
| robots.txt token | meta-webindexer |
| Follows robots.txt | Yes, by the operator's account; Crawl-delay not documented |
| Sends visitors | Yes |
| Links to your pages | Yes |
| Can end up in AI training | Not documented |
| Verification |
None published
Our tracker checks the operator's network (ASN). |
| Our call |
Publisher: Allow E-commerce: Allow Expert site: Allow What the profiles mean |
How to recognize it
User agents, as they reach our logs:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Like Meta-ExternalAgent, it rotates browser prefixes (Chrome, Edge and Firefox) and keeps the tail. Match on meta-webindexer.
- robots.txt token:
meta-webindexer - IP list: none published. We check Meta’s network, AS32934.
- Reverse DNS: none published.
What it does
Meta says Meta-WebIndexer navigates the web to improve the quality of Meta AI’s search results, and that allowing it helps Meta cite and link your content in Meta AI’s responses.
What you gain by allowing it
- Citations with links in Meta AI’s answers.
What you lose
By allowing it
- Crawl load: 827 Meta requests in 8 days on our two sites, more than half of them on the static site.
By blocking it
- Meta AI’s search results stop citing your pages. Training is unaffected either way; that is Meta-ExternalAgent.
How to verify it
Check that the IP belongs to Meta’s network, AS32934. Cloudflare flags the real crawler as a verified bot, but in the category “Page Preview”, as if it built link previews. In our data the flag matched the ASN check on every request.
Gotchas and history
- Cloudflare’s category: “Page Preview” understates it. A rule that blocks “AI Search” bots by Cloudflare category will miss this one.
- No robots.txt reads: it did not request robots.txt once in 8 days.
What we observed
Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).
- 1,516 requests carried a Meta-WebIndexer user agent:
- 827 (55%) came from Meta’s network (AS32934);
- 686 of the 689 others came from the Google Cloud scanner;
- Cloudflare’s flag matched the ASN check on all 1,516.
- The 827 Meta requests covered 622 different paths:
- 708 went to the static content site;
- 311 got a redirect and 104 an error, all of the errors on the static site (93 of them 404s);
- none asked for
/robots.txt.
About Meta
- meta-externalagent crawls for uses such as training Meta’s foundation AI models.
- meta-webindexer crawls to improve Meta AI’s search results, and allowing it helps Meta AI cite and link your content.
- meta-externalfetcher fetches individual links at a user’s request, and may bypass robots.txt.
- facebookexternalhit builds link previews when a page is shared, and may bypass robots.txt for security or integrity checks.
- meta-externalads crawls for Meta’s advertising and business products. We have not seen it in our logs.
The training and search crawlers have separate tokens. To be cited by Meta AI without feeding training, allow meta-webindexer and disallow meta-externalagent.
Meta publishes no IP list or reverse DNS scheme on its crawler page. It calls IP allowlisting the more secure option, but refers site owners to [email protected] for the addresses. A check against Meta’s network, AS32934, proves a request came from Meta, but not which of these agents sent it.