Bot catalog

Checks the quality of Google Ads landing pages. Ignores the robots.txt group for all bots; the mobile variant is listed as retired but still crawls.

User agents, as they reach our logs:

AdsBot-Google (+http://www.google.com/adsbot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)

• First string: the desktop crawler.
• Second string: the mobile one. Google’s page lists it both as active and among retired crawlers, and it made almost half the requests on our sites.

Match on AdsBot-Google, which also catches the mobile variant.

• robots.txt tokens: AdsBot-Google and AdsBot-Google-Mobile
• IP list (JSON): special-crawlers.json
• Reverse DNS: rate-limited-proxy-*.google.com

Full card: what it does, what you gain or lose, what we saw

Ahrefs' crawler for its link and content index, which also powers the Yep search engine. Separate from AhrefsSiteAudit.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Match on the token AhrefsBot. Site Audit is a different bot with its own token, AhrefsSiteAudit: Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit).

• robots.txt token: AhrefsBot
• IP lists (JSON): crawler-ip-ranges (CIDR ranges) and crawler-ips (single addresses)
• Reverse DNS: the hostname always ends in ahrefs.com or ahrefs.net.

Full card: what it does, what you gain or lose, what we saw

Ahrefs' Site Audit crawler, which a customer points at a site. Obeys robots.txt unless the site's verified owner turns that off. Max 30 URLs a minute.

User agent, as it reaches our logs (identical to Ahrefs’ page):

Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

Match on AhrefsSiteAudit, separately from AhrefsBot.

• robots.txt token: AhrefsSiteAudit
• IP list (JSON): crawler-ip-ranges, shared with AhrefsBot.
• Reverse DNS: names ending in ahrefs.com or ahrefs.net.

On our sites in 19-26 October 2026 it made 38 requests, from 35 addresses that are all on Ahrefs’ list, and Cloudflare flagged every one. Firewall rules on both sites answered all 38 with a 403. That includes audits we run ourselves, which a rule matching “Ahrefs” breaks along with AhrefsBot.

Amazon's crawler for improving its products and services, which may include training its AI models. Honours robots.txt and noarchive, ignores Crawl-delay.

User agent, as it reaches our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

Match on Amazonbot. Amazon’s two other agents, Amzn-SearchBot and Amzn-User, use the same pattern with their own names.

• robots.txt token: Amazonbot
• IP list: ip-addresses, an HTML page, not a JSON file.
• Reverse DNS: not documented by Amazon. In practice, names under crawl.amazonbot.amazon forward-confirm, and our trackers accept either the list or that check.

Full card: what it does, what you gain or lose, what we saw

Apple's crawler for Siri, Spotlight and Safari search. It renders pages, follows Googlebot's rules when robots.txt has none for it, and ignores Crawl-delay.

User agent, as it reaches our logs:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

A full Safari string with (Applebot/0.1; ...) at the end. Apple says the browser version in it may change. Match on Applebot, but not Applebot-Extended.

• robots.txt token: Applebot
• IP list (JSON): applebot.json
• Reverse DNS: names under applebot.apple.com.

Full card: what it does, what you gain or lose, what we saw

A robots.txt token, not a crawler. It decides whether content Applebot crawls may train Apple's foundation models. It does not affect Apple's search.

You will not see it in logs. Apple says Applebot-Extended does not crawl webpages; it only changes how data Applebot collects may be used. Any request whose user agent contains Applebot-Extended is fake.

• robots.txt token: Applebot-Extended
• IP list: none, because no request carries the name.
• Reverse DNS: none, for the same reason.

Full card: what it does, what you gain or lose, what we saw

Awario's crawler for brand monitoring, which finds mentions of its customers' brands. Identified by user agent only; Awario publishes no addresses.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)

Awario lists this string next to AwarioSmartBot/1.0 and AwarioRssBot/1.0, but does not say which crawler sends it, and its robots.txt examples name only the other two.

• robots.txt token: none documented for AwarioBot. Use AwarioSmartBot and AwarioRssBot, which Awario does document.
• IP list: none. Awario says not to identify its bots by IP.
• Reverse DNS: none.

Awario says its bots slow to about one request every 3 seconds, or less if robots.txt asks, and do not re-request pages they already have. On our sites in 19-26 October 2026 it made 161 requests, all to the WordPress site, from Hetzner’s network, on 3 days. Cloudflare did not flag them, and the site’s firewall rule for SEO crawlers did not catch this name: 160 got through.

Baidu's search crawler. One token covers all of Baidu's agents; it ignores Crawl-delay, and Baidu publishes no IP list on purpose.

User agents, as they reach our logs (desktop, then mobile):

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Baidu’s own pages print these with slightly different spacing, and one has a full-width bracket. Match on Baiduspider, after the more specific Baiduspider-render. Baidu also runs -image, -video, -news, -favo, -cpro and -ads variants.

• robots.txt token: Baiduspider
• IP list: none. Baidu says its ranges change and it cannot publish them.
• Reverse DNS: names under baidu.com or baidu.jp, such as baiduspider-111-206-198-69.crawl.baidu.com, confirmed forward.

Full card: what it does, what you gain or lose, what we saw

Baidu's rendering crawler, which loads CSS, JavaScript and images to understand pages. Covered by the Baiduspider token.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)

Baidu also documents an iPhone variant and one for its mini-programs (Smartapp). Match on Baiduspider-render before the plain Baiduspider.

• robots.txt token: Baiduspider (no separate one documented).
• IP list: none.
• Reverse DNS: names under baidu.com or baidu.jp, confirmed forward, as for Baiduspider.

On our sites in 19-26 October 2026 it made 11 requests, all verified by reverse DNS and flagged by Cloudflare. Our firewall answered all 11 with a 403 at the edge.

Babbar's crawler for the link graph behind its SEO tools. Obeys robots.txt and Crawl-delay; its reverse DNS domain is babbar.eu, not babbar.tech.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)

Match on Barkrowler. Babbar’s robots.txt examples write the token in lower case, barkrowler.

• robots.txt token: barkrowler
• IP list (JSON): barkrowler-ip-ranges.json, two /24 ranges, dated October 2024.
• Reverse DNS: names under babbar.eu.

It waits 5 seconds between requests to the same host by default, and keeps recrawling URLs that answered 301 or 404. On our sites in 19-26 October 2026 it made 308 requests flagged by Cloudflare, on 3 days, almost all on the static site. The WordPress site’s firewall blocked the 3 that reached it, all for robots.txt.

Microsoft's main crawler for Bing, whose index also feeds Copilot answers. Honours robots.txt and Crawl-delay.

User agents, as they reach our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

• First string: 98% of verified requests. Microsoft says the version number tracks the latest Edge, but Chrome 116 is a 2023 version.
• Second string: two complete user agents glued together. It is real Bing traffic, verified by reverse DNS.
• Short form in Microsoft’s docs: absent from our logs in these 8 days:

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Match on bingbot. Microsoft’s BingPreview snapshots use the same bingbot string, so they count as Bingbot too.

• robots.txt token: bingbot
• IP list (JSON): bingbot.json
• Reverse DNS: names ending in search.msn.com, such as msnbot-157-55-33-18.search.msn.com.

Full card: what it does, what you gain or lose, what we saw

Brave Search's crawler. Brave says it has no separate user agent and follows Googlebot's permissions, but it announces itself as Bravebot.

User agents, as they reach our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 (compatible; Bravebot/1.0; +https://brave.com/search/)

• First string: the real one, flagged by Cloudflare. It sends Chrome/W.X.Y.Z literally, a template nobody filled in, and links to Brave’s crawler help page.
• Second string: what the Google Cloud scanner sent. It is a plausible fake with a different link.

Match on Bravebot, then tell them apart by the link and the network.

• robots.txt token: none documented. Brave’s crawler follows your Googlebot rules.
• IP list: none published. The real requests came from Hetzner addresses.
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

BrightEdge's crawler for its SEO platform's audits and recommendations. BrightEdge's documentation could not be checked for this card.

User agent, as it reached our logs:

BrightEdge Crawler/1.0 ([email protected])

Match on BrightEdge.

• robots.txt token: not confirmed. BrightEdge’s support page returned a 403 to our tools, and secondary listings disagree between BrightEdge and BrightEdge Crawler.
• IP list: not confirmed. The support page reportedly mentions IP allowlisting.
• Reverse DNS: not confirmed.

On our sites in 19-26 October 2026 it made a single request to the static site, which Cloudflare flagged as a verified bot in “Advertising & Marketing”. This card needs a manual check of BrightEdge’s page.

ByteDance documents Bytespider as the crawler of Toutiao Search. The Bytespider strings most logs show are not the ones it documents.

User agents as ByteDance documents them (desktop and Android; an iOS variant also exists):

Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)

What reached our logs instead:

Mozilla/5.0 (compatible; Bytespider; [email protected]) AppleWebKit/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])

The documented strings link to Toutiao’s webmaster site. The ones in our logs, and in most published bot lists, carry [email protected] or a document link instead. Match on Bytespider, then check the network.

• robots.txt token: Bytespider
• IP list: none as a file. ByteDance prints ten /24 ranges on its page, including 110.249.201.0/24, 111.225.148.0/24 and 220.243.135.0/24.
• Reverse DNS: names under bytedance.com, such as bytespider-111-225-148-148.crawl.bytedance.com. ByteDance describes only the reverse lookup; confirm it forward anyway.

Full card: what it does, what you gain or lose, what we saw

Common Crawl's crawler. Its open web archive is one of the most used sources of training data for large language models. Obeys robots.txt and Crawl-delay.

User agent, as Common Crawl documents it (the scanner sent the same string):

CCBot/2.0 (https://commoncrawl.org/faq/)

Match on CCBot; Common Crawl says the version number may rise. It also warns that other crawlers falsely claim to be CCBot.

• robots.txt token: CCBot
• IP list (JSON): ccbot.json
• Reverse DNS: names under crawl.commoncrawl.org, IPv4 only, confirmed forward. Its IPv6 range has no reverse DNS, so use the list for those addresses.

Full card: what it does, what you gain or lose, what we saw

A name in bot lists that Zhipu AI does not use. Zhipu documents its training crawler as GLMBot. Every ChatGLM-Spider request on our sites was fake.

User agent, as the scanner sent it to our sites:

Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://zhipuai.cn/)

Zhipu’s public training summary for its GLM-5 models names its crawler GLMBot, not ChatGLM-Spider, and gives no full user agent. A block for ChatGLM-Spider may be aimed at a name Zhipu does not use.

• robots.txt token: none documented under this name. To refuse Zhipu’s training crawler, the documented name is GLMBot.
• IP list: none.
• Reverse DNS: none.

On our sites in 19-26 October 2026, all 742 requests with this name came from the Google Cloud scanner, and none used GLMBot.

Fetches a page when a ChatGPT or custom GPT user asks for it. OpenAI says robots.txt may not apply, and it does not decide what appears in ChatGPT search.

User agent, as it reaches our logs (identical to OpenAI’s docs):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Match on the token ChatGPT-User. On our sites 98% of requests with this user agent were fake, so always check the IP.

• robots.txt token: ChatGPT-User
• IP list (JSON): chatgpt-user.json
• Reverse DNS: none published by OpenAI.

Full card: what it does, what you gain or lose, what we saw

Asks for /.well-known/traffic-advice before Chrome prefetches your page from Google Search results through a private proxy. Not a crawler.

User agent, as it reaches our logs:

Chrome Privacy Preserving Prefetch Proxy

It asks for one path only, /.well-known/traffic-advice. The prefetches themselves look like normal Chrome page requests with the header Sec-Purpose: Prefetch; anonymous-client-ip, coming from the proxy instead of the user.

• robots.txt token: none; the control is the traffic-advice file.
• IP list: none published. Our requests came from Google’s own network (AS15169).
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

The web fetch of Anthropic's coding tool. Requests leave from the developer's own machine, so no IP list can verify them.

User agent, as it reaches our logs:

Claude-User (claude-code/2.1.308; +https://support.anthropic.com/)

The version after claude-code/ changes with every release. Match on claude-code/ before Claude-User, or these requests land in the Claude-User class.

• robots.txt token: none documented.
• IP list: none. The fetch leaves from the developer’s own network, so it shows up from home and mobile ISPs. On our sites, 3 requests in 19-26 October 2026 came from two consumer ISPs.
• Reverse DNS: none.

Indexes pages to improve Claude's search results. On our sites every request with this name in October 2026 was fake.

User agent, as the scanner sent it to our sites:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])

We have not seen a verified request, and Anthropic’s Help Center gives the name but not the full string. So treat this as the shape to match, not proof. Match on the token Claude-SearchBot.

• robots.txt token: Claude-SearchBot
• IP list (JSON): bots.json
• Reverse DNS: none published by Anthropic.

Full card: what it does, what you gain or lose, what we saw

Fetches a page when a Claude user asks about it. Requests from Claude's API come from a range that bots.json does not list.

User agent, as it reaches our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected])

Match on the token Claude-User, but check for claude-code/ first. Claude Code’s fetches use Claude-User (claude-code/...) and are a different agent.

• robots.txt token: Claude-User
• IP list (JSON): bots.json
• Also from Anthropic: API tools (web fetch, web search, MCP) call out from 160.79.104.0/21, a range published on Anthropic’s API IP addresses page but not in bots.json.
• Reverse DNS: none published by Anthropic.

Full card: what it does, what you gain or lose, what we saw

Anthropic's crawler for content that may train its models. Honours robots.txt and Crawl-delay. Separate from Claude's search and user fetches.

User agents, as they reach our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; ClaudeBot/1.0; [email protected])

• First string: the plain crawler, which sent 97% of the verified requests.
• Second string: a Chrome-based variant. It also fetches stylesheets, scripts and images, the way a browser loads a page.

Match on the token ClaudeBot. The old names anthropic-ai and Claude-Web are not in Anthropic’s current docs; any request using them is fake.

• robots.txt token: ClaudeBot
• IP list (JSON): bots.json, one list for all three of Anthropic’s named bots.
• Reverse DNS: none published by Anthropic.

Full card: what it does, what you gain or lose, what we saw

A name in bot lists, not a documented Cohere crawler. Cohere says it does not crawl the web to train its models. Every request here was fake.

User agent, as the scanner sent it to our sites:

Mozilla/5.0 (compatible; cohere-ai; +https://cohere.com/crawler)

The link in it returns a 404. Cohere’s crawler page lists no bots at all, and says Cohere does not currently crawl the web to train generative AI models.

• robots.txt token: none documented.
• IP list: none.
• Reverse DNS: none.

On our sites in 19-26 October 2026, 807 of the 809 requests with this name came from the Google Cloud scanner. The other 2 came from unrelated networks, which makes them fakes as well.

Criteo's crawler for contextual ads. It categorises a page only when Criteo gets an ad request for it. Its addresses are in three documented networks.

User agent, as it reaches our logs:

CriteoBot/0.1 (+https://www.criteo.com/criteo-crawler/)

Match on CriteoBot.

• robots.txt token: CriteoBot/0.1, as written in Criteo’s examples.
• IP list: none as a file. Criteo names three networks (AS44788, AS19750, AS55569) and their ranges on its page.
• Reverse DNS: none published.

On our sites in 19-26 October 2026 it made 238 requests to the static site, which shows ads, from all three of Criteo’s networks. Cloudflare flagged them as “Advertising & Marketing”. It covered 186 pages and asked for robots.txt 12 times.

DataForSEO's crawler for the backlink database it sells through its API to SEO tools. Verifiable by reverse DNS; a separate API crawler can ignore robots.txt.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)

DataForSEO’s page shows the string without /1.0 and lists the version separately. Match on DataForSeoBot.

• robots.txt token: DataForSeoBot
• IP list: none as a file. DataForSEO prints three IPv4 subnets on its page, such as 136.243.228.176/29, plus three IPv6 ones.
• Reverse DNS: names like crawling-gateway-136-243-228-176.dataforseo.com.

Full card: what it does, what you gain or lose, what we saw

A name in bot lists. DeepSeek admits to a training crawler that respects robots.txt but never names it. Every request with this name on our sites was fake.

User agent, as the scanner sent it to our sites:

Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/)

DeepSeek’s public training summary says it runs a crawler that respects robots.txt, but gives it no name or user agent. Without a documented token there is nothing exact to write in robots.txt.

• robots.txt token: none documented.
• IP list: none.
• Reverse DNS: none.

On our sites in 19-26 October 2026, all 710 requests with this name came from the Google Cloud scanner.

Diffbot's crawler for its Knowledge Graph and web search. Not used for AI training, but robots.txt can be overridden by agreement.

User agent, as it reached our logs:

Mozilla/5.0 (compatible; Diffbot/1.0; +https://www.diffbot.com)

Diffbot documents only the tokens Diffbot and Diffbot-User, not full strings. Match on the token. Jobs that Diffbot’s customers run can carry any user agent the customer sets, so not all Diffbot traffic says Diffbot.

• robots.txt token: Diffbot
• IP list: none.
• Reverse DNS: none.

On our sites in 19-26 October 2026 it made a single request, not flagged by Cloudflare, so we cannot say whether it was Diffbot.

Moz's crawler for its link index. Moz's documentation could not be checked for this card, so only what our logs show is stated.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected])

The link points to Open Site Explorer, Moz’s former name for its link tool, and the contact is [email protected]. Match on DotBot.

• robots.txt token: DotBot
• IP list: not checked. Moz’s site refuses the tools we use to read documentation, so this card states nothing from Moz’s pages. It needs a manual check in a browser.
• Reverse DNS: not checked, for the same reason.

On our sites in 19-26 October 2026 it made 339 requests from 3 addresses in one network (AS23033), flagged by Cloudflare as “Search Engine Optimization”. The static site answered all of them. The WordPress site’s firewall blocked the 2 that reached it, both for robots.txt.

Fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. DuckDuckGo says the data does not train AI models.

User agents, the real one first, then the scanner’s:

DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
DuckAssistBot/1.1 (https://duckduckgo.com/duckassistbot)

DuckDuckGo documents version 1.2 with an http://...html link. The scanner on our sites sent version 1.1 with a different link and no semicolon. Match on DuckAssistBot, then check the IP.

• robots.txt token: DuckAssistBot
• IP list (JSON): duckassistbot.json
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

DuckDuckGo's own crawler, used to improve its search results. Its IP list is the same file as DuckAssistBot's.

User agents, as they reach our logs:

DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'
DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)

• First string: the only one DuckDuckGo documents.
• Second string: includes the single quotes, and it was the most common of the three. A parser that strips or rejects quoted values miscounts it.
• Version 1.0: also turned up.

Match on DuckDuckBot, which covers the -Https variant.

• robots.txt token: DuckDuckBot
• IP list (JSON): duckduckbot.json
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

ElevenLabs' crawler for content that may train its generative AI models. On our sites it sent more requests than any other crawler, almost all in one day.

User agent, as it reaches our logs (identical to ElevenLabs’ docs):

Mozilla/5.0 (compatible; ElevenlabsCrawler/1.0; +https://elevenlabs.io/docs/bots)

Match on ElevenlabsCrawler. Its sibling ElevenlabsBot/1.0 is a different agent with its own token.

• robots.txt token: ElevenlabsCrawler
• IP list: none published.
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

Exa's crawler for its search API, which AI apps query for web results. Without a group of its own it follows your rules for major search engines.

User agent, as it reaches our logs (identical to Exa’s docs):

Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)

Match on ExaSearchBot. Exa’s user-triggered fetcher is ExaUser/1.0, which Exa says may not follow robots.txt. “Exabot” is an unrelated name.

• robots.txt token: ExaSearchBot
• IP list: none. Exa signs requests with Web Bot Auth instead; its keys are at crawler.exa.ai/.well-known/http-message-signatures-directory.
• Reverse DNS: none published.

On our sites in 19-26 October 2026 it made 172 requests on 131 paths, all flagged by Cloudflare as “Search Engine Crawler”, from Latitude.sh and ColoCrossing networks. 11 were for /robots.txt. The scanner did not use this name.

Builds the preview card when someone shares your link in Meta's apps. Its name is also borrowed by Apple's Messages app for previews built on users' devices.

User agents, as they reach our logs:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

• First string: Meta’s crawler.
• Second string: not Meta. It is the user agent that Apple’s Messages app is widely reported to send when it builds a link preview on the user’s own device. It names Facebook’s and Twitter’s bots so that sites hand it their preview tags.

Match on facebookexternalhit, then check the network.

• robots.txt token: facebookexternalhit
• IP list: none published. We check Meta’s network, AS32934.
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

Fetches RSS and Atom feeds for Google News and WebSub. Google lists it with user-triggered fetchers, but it arrives from the special-case crawlers' addresses.

User agent, as it reaches our logs:

FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)

Match on FeedFetcher-Google. Check for Feedspot first: its user agent ends with “like FeedFetcher-Google” and is not Google.

• robots.txt token: none documented.
• IP list (JSON): user-triggered-fetchers-google.json. Google publishes the group in three files.
• Reverse DNS: Google documents *.gae.googleusercontent.com or google-proxy-*.google.com for this group.

On our sites in 19-26 October 2026 it made 8 requests for feed URLs. All 8 resolved to rate-limited-proxy-*.google.com, the pattern of Google’s special-case crawlers, not the one documented for its group. A verifier that follows Google’s per-group patterns strictly would reject the real Feedfetcher. Cloudflare flagged all 8 as “Feed Fetcher”.

The feed fetcher of the Feedly reader. It acts for subscribed users, does not follow robots.txt, and reports the subscriber count in its user agent.

User agent, as it reaches our logs:

Feedly/1.0 (+http://www.feedly.com/fetcher.html; 53 subscribers; )

The number is how many Feedly users follow the feed, which makes Feedly’s requests the cheapest audience measurement a blog has. Match on Feedly.

• robots.txt token: none; robots.txt is not followed.
• IP list: none. Feedly says its addresses change and to identify it by user agent.
• Reverse DNS: none.

Feedly fetches most feeds at most about once an hour, and once a day if the feed announces a push hub. On our sites in 19-26 October 2026 it made 89 requests, all for the WordPress site’s /feed/. Cloudflare flagged 83 of them as “Feed Fetcher”. The user agent reported 53 subscribers.

The fetcher of the Feedspot reader and feed directory. Its user agent ends with 'like FeedFetcher-Google', but it is not Google.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)

The string ends with like FeedFetcher-Google. Match on Feedspot before FeedFetcher-Google, or Feedspot is counted as a spoofed Google bot.

• robots.txt token: not checked.
• IP list: not checked. Feedspot’s fetcher page returned a 403 to our tools, so this card needs a manual check.
• Reverse DNS: not checked.

On our sites in 19-26 October 2026 it made 2 requests, both for one article rather than the feed, from Amazon’s network. Cloudflare did not flag them.

An ads crawler from Google's network that Cloudflare verifies, but that Google's crawler documentation does not list.

User agent, as it reaches our logs:

Google-Display-Ads-Bot

• robots.txt token: none documented.
• IP list: none. Google’s crawler pages, including the special-case list, do not mention this name.
• Reverse DNS: none documented.

On our sites in 19-26 October 2026 it made 2 requests, both for the homepage of the site that shows AdSense ads. They came from Google’s network (AS15169), and Cloudflare flagged them as a verified bot in “Advertising & Marketing”. Cloudflare knows the bot, but Google does not document it. Third-party bot lists tie it to AdSense site checks.

A robots.txt token, not a crawler. It decides whether content Google crawls may train Gemini and ground Gemini Apps and Vertex AI. It does not affect Search.

You will not see it in logs. Google says Google-Extended has no user agent string of its own; the crawling is done by Google’s existing crawlers, mostly Googlebot. Any request whose user agent contains Google-Extended is fake.

• robots.txt token: Google-Extended
• IP list: none, because no request carries the name.
• Reverse DNS: none, for the same reason.

Full card: what it does, what you gain or lose, what we saw

Google's abuse crawler, which checks links posted on Google properties for malware. It ignores robots.txt by design.

User agent, as it reaches our logs:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.36 Mobile Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)

Google documents the name as just Google-Safety; in practice it arrives inside a full mobile Chrome string. Match on the token.

• robots.txt token: none.
• IP list (JSON): special-crawlers.json
• Reverse DNS: rate-limited-proxy-*.google.com

On our sites in 19-26 October 2026 it made 19 requests on one day, loading a page together with its stylesheets, scripts and fonts. All 19 passed forward-confirmed reverse DNS. Cloudflare did not flag them as a verified bot, so a rule that admits only Cloudflare-verified bots would block Google’s malware check.

Google's main crawler. Fetches pages for Google Search, including Discover, Images, News and the AI features in Search.

User agents, as they reach our logs:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• First string: the smartphone crawler, which fetches the pages. The Chrome version varies; we saw four different ones in 8 days.
• Second, short string: it made every robots.txt request in our data.
• Desktop crawler: documented by Google, but absent from our logs in those 8 days:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Match on the token Googlebot, then verify the IP. The user agent alone proves nothing.

• robots.txt token: Googlebot
• IP list (JSON): common-crawlers.json
• Reverse DNS: crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com

Full card: what it does, what you gain or lose, what we saw

Google's image crawler, for Google Images, Discover, Video mode and the images, logos and favicons shown in Search.

User agent, as it reaches our logs:

Googlebot-Image/1.0

Match on Googlebot-Image before the plain Googlebot, or its requests are counted as Googlebot.

• robots.txt token: Googlebot-Image
• IP list (JSON): common-crawlers.json
• Reverse DNS: crawl-*.googlebot.com, the same as Googlebot.

On our two sites in 19-26 October 2026 it made 43 requests on all 8 days. Every one passed forward-confirmed reverse DNS, and Cloudflare flagged it as “Search Engine Crawler”. 37 of the requests were for image files.

Google's generic crawler for product teams. Google says it affects no specific product; Cloudflare files it as an AI crawler.

User agents, as they reach our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/153.0.8010.52 Safari/537.36
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; GoogleOther)
GoogleOther

• First and second strings: the desktop and mobile crawlers.
• Third, bare GoogleOther: made every robots.txt request we saw, plus a few homepage requests.

Match on the token GoogleOther, after the more specific GoogleOther-Image and GoogleOther-Video.

• robots.txt token: GoogleOther
• IP list (JSON): common-crawlers.json
• Reverse DNS: crawl-*.googlebot.com, the same as Googlebot.

Full card: what it does, what you gain or lose, what we saw

The image version of GoogleOther, Google's generic crawler for product teams. Google says it affects no specific product.

User agent, as it reaches our logs:

GoogleOther-Image/1.0

Match on GoogleOther-Image before the plain GoogleOther.

• robots.txt token: GoogleOther-Image
• IP list (JSON): common-crawlers.json
• Reverse DNS: crawl-*.googlebot.com

It made a single verified request on our sites in 19-26 October 2026. Like GoogleOther, Cloudflare flags it as “AI Crawler”.

OpenAI's crawler for training its foundation models. It does not feed ChatGPT search, which crawls as OAI-SearchBot.

User agent, as it reaches our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)

Match on the token GPTBot; the version number changes. Then verify the IP: four out of five requests with this user agent on our sites were fake.

• robots.txt token: GPTBot
• IP list (JSON): gptbot.json
• Reverse DNS: none published by OpenAI.

The lists for OpenAI’s other agents (OAI-SearchBot, ChatGPT-User, OAI-AdsBot) are on the OpenAI card.

Full card: what it does, what you gain or lose, what we saw

Integral Ad Science's crawler, which rates pages for brand safety. IAS says blocking it makes the site inaccessible to its partner advertisers.

User agent, as it reaches our logs:

IAS Crawler (ias_crawler; http://integralads.com/site-indexing-policy/)

Match on ias_crawler; the second IAS bot announces itself as ias_wombles.

• robots.txt token: ias_crawler
• IP list (JSON): iasbot.json. Note that it uses publicIPs with single ipv4 entries, not the usual prefixes format.
• Reverse DNS: none published.

IAS says it reads robots.txt before every crawl. On our sites in 19-26 October 2026 it made 58 requests to the static site, which shows ads, from Amazon’s network. Cloudflare flagged 39 of them. 19 were for robots.txt.

The feed fetcher of the Inoreader reader. It fetches only feeds its users subscribe to, ignores robots.txt, and states the subscriber count in its user agent.

User agent, as it reaches our logs:

Inoreader/1.0 (+http://www.inoreader.com/feed-fetcher; 1 subscribers; )

The number is how many Inoreader users subscribe to the feed. Inoreader says it sometimes fetches with a standard browser user agent instead, and may retry failed requests from another address. Match on Inoreader.

• robots.txt token: none; robots.txt is not read.
• IP list: ip_list.txt, plain text with CIDR ranges, not JSON.
• Reverse DNS: none published.

It polls active feeds every 30 minutes to a few hours, and less often as a feed goes quiet. On our sites in 19-26 October 2026 it made 163 requests, all for the WordPress site’s /feed/, all flagged by Cloudflare as “Feed Fetcher”. The user agent reported 1 subscriber.

The crawler of Ison Search, an AI search service from Indonesia. Obeys robots.txt and Crawl-delay, and publishes an IP list.

User agent, as it reaches our logs (identical to Ison’s page):

IsonSearchBot/1.0 (+https://search.isonai.com/bot)

Ison says it uses this one user agent for every request, headless renders included. Match on IsonSearchBot.

• robots.txt token: IsonSearchBot
• IP list (JSON): ips.json. Ison notes that one node uses an unpublished shared mobile address, so an IP outside the list is not proof of a fake. It also signs requests with Web Bot Auth.
• Reverse DNS: Ison says not to rely on it.

On our sites in 19-26 October 2026 it made 3 requests, all flagged by Cloudflare; 2 were for /robots.txt.

Keenable's crawler for an index that answers AI agents with cited sources. Follows RFC 9309 and publishes an IP list.

User agent, as it reaches our logs (identical to Keenable’s docs):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/bot)

Match on KeenableBot. Keenable also honours noindex, nofollow and none in meta robots and X-Robots-Tag.

• robots.txt token: KeenableBot
• IP list (JSON): keenablebot.json
• Reverse DNS: none published.

On our sites in 19-26 October 2026 it made 67 requests on 3 hostnames, all flagged by Cloudflare as “Search Engine Crawler”, from Google Cloud addresses. 19 were for /robots.txt.

Moonshot AI's crawler for Kimi's search index. Obeys robots.txt and Crawl-delay. Every request with its name on our sites was fake.

User agent as Moonshot documents it, then the one the scanner sent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Kimi-SearchBot/1.0; +https://www.kimi.com/policies/kimi-crawlers
Mozilla/5.0 (compatible; Kimi-SearchBot/1.0; +https://kimi.ai/)

Match on Kimi-SearchBot; the short form with a bare kimi.ai link is the scanner’s.

• robots.txt token: Kimi-SearchBot
• IP list (JSON): kimi-searchbot.json, identical to the KimiBot and Kimi-User lists.
• Reverse DNS: none published.

On our sites in 19-26 October 2026, all 758 requests with this name came from the Google Cloud scanner. Kimi has few users outside China; see Almost nobody in the West uses Kimi or Doubao.

Moonshot AI's crawler for content that may train Kimi's models. Obeys robots.txt and Crawl-delay. Every request with its name on our sites was fake.

User agent as Moonshot documents it, then the one the scanner sent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; KimiBot/1.0; +https://www.kimi.com/policies/kimi-crawlers
Mozilla/5.0 (compatible; KimiBot/1.0; +https://kimi.ai/)

The real string links to Moonshot’s crawler policy; the short one with a bare kimi.ai link is the scanner’s. Match on KimiBot.

• robots.txt token: KimiBot
• IP list (JSON): kimibot.json. It is byte for byte the same file as the Kimi-SearchBot and Kimi-User lists: four addresses that prove “Kimi” but not which agent.
• Reverse DNS: none published.

On our sites in 19-26 October 2026, all 719 requests with this name came from the Google Cloud scanner. Moonshot warns that blocking its addresses may stop it from reading your latest robots.txt.

Google's page-quality audit. Requests from Google's network are PageSpeed Insights tests that someone ran on your page, not necessarily you.

User agent, as it reaches our logs:

Mozilla/5.0 (Linux; Android 11; moto g power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Mobile Safari/537.36 Chrome-Lighthouse

A full mobile Chrome string ending in Chrome-Lighthouse. Match on that token. The same token appears when anyone runs Lighthouse on their own machine; only requests from Google’s network are PageSpeed Insights.

• robots.txt token: none.
• IP list: none published for PageSpeed Insights. Our requests came from Google’s own network (AS15169).
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

Linkup's crawler for the search index that grounds AI applications. Obeys robots.txt and publishes its addresses as a plain-text list.

User agent, as it reaches our logs:

LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; [email protected])

Linkup’s example uses https://www.linkup.so/bot; our logs show the link without www. Match on LinkupBot. linkup.com is a different company with a crawler of a similar name.

• robots.txt token: LinkupBot
• IP list: linkupbot-ips.txt, plain text with CIDR ranges, not JSON.
• Reverse DNS: none published.

On our sites in 19-26 October 2026 it made 78 requests, all to the WordPress site, all flagged by Cloudflare. 37 of them (47%) were for /robots.txt.

The crawler of Mediaboard, a media-monitoring service. Mediaboard publishes no crawler documentation; on our site it mostly read the feed.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; mediaboardbot/1.0; +https://www.mediaboard.com/)

Match on mediaboardbot.

• robots.txt token: none documented. Mediaboard’s site has no crawler page.
• IP list: none.
• Reverse DNS: none.

On our sites in 19-26 October 2026 it made 377 requests to the WordPress site, from 47 Amazon addresses, all flagged by Cloudflare as “Monitoring & Analytics”. 324 were for /feed/, and 7 each for robots.txt and a sitemap.

The AdSense crawler. It reads pages on sites that show Google ads, to pick relevant ads, and ignores the robots.txt group for all bots.

User agents, as they reach our logs:

Mediapartners-Google
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Mediapartners-Google) Chrome/153.0.8010.36 Safari/537.36
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Mediapartners-Google) Chrome/153.0.8010.36 Mobile Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko, Mediapartners-Google) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION]

• First string: the bare desktop name.
• Second and third: the desktop and mobile variants. Note that the token sits inside the parentheses after KHTML, like Gecko.
• Last string: sent with its template placeholders unfilled. Do not treat that as a sign of a fake; the real crawler does it.

Match on the token Mediapartners-Google.

• robots.txt token: Mediapartners-Google
• IP list (JSON): special-crawlers.json
• Reverse DNS: rate-limited-proxy-*.google.com according to Google, but real requests also come from crawl-*.googlebot.com.

Full card: what it does, what you gain or lose, what we saw

Determ's media-monitoring crawler, formerly Mediatoolkit. It checks pages for phrases customers track; on our site it polled the same pages every half hour.

User agent, as it reaches our logs:

Mediatoolkitbot ([email protected])

No browser prefix and no version, just the name and a complaints address under the old brand. Determ’s page documents the token, not a full string. Match on Mediatoolkitbot.

• robots.txt token: Mediatoolkitbot
• IP list: none published.
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

Meta's crawler for uses such as training its foundation AI models. Its user agent looks like a full desktop browser with the token at the end.

User agents, as they reach our logs:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

In 8 days it used 11 different browser prefixes: Chrome, Edge and Firefox on Windows, Mac and Linux. Only the (compatible; meta-externalagent/1.1 ...) tail stays the same, so match on meta-externalagent. Meta’s docs show the bare form, meta-externalagent/1.1, with a relative link.

• robots.txt token: meta-externalagent
• IP list: none published. Meta’s crawler page says IP allowlisting is more secure but points to its documentation index or [email protected] for the addresses. We check Meta’s own network, AS32934.
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

Fetches individual links when a user asks a Meta AI product to. Meta says it may bypass robots.txt. On our sites every request with this name was fake.

User agent as Meta documents it, then the one the scanner sent to our sites:

meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
Mozilla/5.0 (compatible; Meta-ExternalFetcher/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)

The fake says version 1.0 where Meta documents 1.1. We have not seen a real request, so we do not know which full browser prefix the real one carries. Match on meta-externalfetcher, in any case.

• robots.txt token: meta-externalfetcher
• IP list: none published. We check Meta’s network, AS32934.
• Reverse DNS: none published.

All 749 requests with this name on our sites in 19-26 October 2026 came from the Google Cloud scanner, and none from Meta.

Meta's crawler for Meta AI search results. Meta says allowing it helps Meta AI cite and link your content.

User agents, as they reach our logs:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Like Meta-ExternalAgent, it rotates browser prefixes (Chrome, Edge and Firefox) and keeps the tail. Match on meta-webindexer.

• robots.txt token: meta-webindexer
• IP list: none published. We check Meta’s network, AS32934.
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

Open-source, self-hosted feed reader. Each copy fetches its own users' feeds, so there is no central operator, IP list or robots.txt policy.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; Miniflux/2.2.12; +https://miniflux.app)

The number is the Miniflux version. Whoever runs a copy can change the user agent globally or per feed, so not all Miniflux traffic says Miniflux. Match on Miniflux.

• robots.txt token: none.
• IP list: none; every copy runs on its owner’s server.
• Reverse DNS: none.

On our sites in 19-26 October 2026 it made 12 requests on 2 days, all for the WordPress site’s /feed/, from one Hetzner address. That is one person’s Miniflux.

Majestic's distributed crawler for its backlink index. It runs on volunteers' machines, has no IP list, and reads a 403 on robots.txt as "disallow all".

User agents, as they reach our logs:

Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; MJ12bot/v2.0.7; http://mj12bot.com/)

Majestic’s page names version 1.4.8, from 2017, as current, yet v2.0.7 is also live. Match on MJ12bot.

• robots.txt token: MJ12bot
• IP list: none. The crawler runs on volunteers’ computers, so its addresses cannot be listed, and Majestic asks site owners not to block it by IP.
• Reverse DNS: none. The only verification is an identification string you arrange with Majestic by email, sent in a CRAWLER-IDENT header or inside the user agent.

Full card: what it does, what you gain or lose, what we saw

Bing's old crawler name. Microsoft's crawler list no longer includes it; the one request we saw was a scanner looking for a .git folder.

User agent, as it reached our logs:

msnbot-media/1.1 ( http://search.msn.com/msnbot.htm)

• robots.txt token: none current. Microsoft’s list of Bing crawlers names Bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview.
• IP list: none for this name.
• Reverse DNS: none for this name. Do not confuse it with real Bingbot, whose reverse DNS names look like msnbot-157-55-33-18.search.msn.com.

Our two sites saw a single request with this name in 19-26 October 2026. It came from a DigitalOcean address and asked for /artykuly/.git/config, so it was a scanner, not Bing.

Feed requests from the NetNewsWire reader app on a Mac or iPhone, sent from the user's own device.

User agent, as it reaches our logs:

NetNewsWire (RSS Reader; https://netnewswire.com/)

The app’s source code also uses a browser-like user agent for icons, homepages and images, so not all of its traffic says NetNewsWire. Match on NetNewsWire.

• robots.txt token: none.
• IP list: none; the app runs on users’ devices.
• Reverse DNS: none.

On our sites in 19-26 October 2026 it made 2 requests for the WordPress site’s /feed/, from a consumer network, on 2 days. That is a reader, not a crawler.

Checks landing pages submitted as ads in ChatGPT. It visits only those pages; on our sites every request with this name was fake.

User agent, as OpenAI documents it:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Match on the token OAI-AdsBot. OpenAI says it only visits pages submitted as ads, so on a site that does not advertise in ChatGPT any request with this name is fake. On our sites all 727 such requests in 19-26 October 2026 came from the Google Cloud scanner. The string matched OpenAI’s documented one exactly.

• robots.txt token: none documented.
• IP list (JSON): adsbot.json
• Reverse DNS: none published by OpenAI.

OpenAI's crawler for ChatGPT search. Sites that block it are not shown in ChatGPT search answers, apart from navigational links.

User agents, as they reach our logs:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

• Browser prefix: all variants start with a full Chrome-on-Mac string, so a rule that looks for “bot” at the start of the user agent misses them.

• robots.txt marker: the second string carries it. OpenAI says robots.txt fetches may use it.

• Versions: 1.0 and 1.4 are both in use.

• Matching: use the token OAI-SearchBot.

• robots.txt token: OAI-SearchBot

• IP list (JSON): searchbot.json

• Reverse DNS: none published by OpenAI.

Full card: what it does, what you gain or lose, what we saw

Fetches a page when a Perplexity user's question needs it, and may link it in the answer. Perplexity says it generally ignores robots.txt.

User agent, as Perplexity documents it (the scanner on our sites sent the same string):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Match on the token Perplexity-User, then check the IP. On our sites not one request with this user agent was real.

• robots.txt token: Perplexity-User, though Perplexity says the fetcher generally ignores it.
• IP list (JSON): perplexity-user.json. The docs link it under perplexity.com, which redirects here.
• Reverse DNS: none published by Perplexity.

Full card: what it does, what you gain or lose, what we saw

Perplexity's crawler for its search results, which link the pages they use. Not a verified bot at Cloudflare since August 2025.

User agent, as it reaches our logs (identical to Perplexity’s docs):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Match on the token PerplexityBot, then check the IP: 88% of requests with this user agent on our sites were fake.

• robots.txt token: PerplexityBot
• IP list (JSON): perplexitybot.json. Perplexity’s docs link it under perplexity.com, which redirects here.
• Reverse DNS: none published by Perplexity.

Full card: what it does, what you gain or lose, what we saw

Huawei's crawler for Petal Search, Huawei Assistant and its AI Search. One token covers all three. Verifiable by reverse DNS.

User agents, as they reach our logs:

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (HTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

The mobile string really says HTML, like Gecko, missing the K that Huawei’s own example has. Match on PetalBot.

• robots.txt token: PetalBot. There is no separate switch for AI Search.
• IP list: none published.
• Reverse DNS: names under petalsearch.com, such as petalbot-114-119-128-10.petalsearch.com, confirmed forward.

On our sites in 19-26 October 2026 it made 155 requests flagged by Cloudflare, in the category “AI Crawler”, from Huawei Cloud (AS136907). Our firewall answered all of them with a 403 at the edge. Our post Almost nobody in the West uses Kimi or Doubao first missed Huawei’s reverse DNS check; its correction of 26 October 2026 adds it.

Pinterest's crawler for Pins: product and article details, prices, broken links. Crawl-delay is capped at 1; Pinterest says larger values hurt distribution.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)

Pinterest also documents Pinterest/0.2 and a mobile string built on Chrome 41. Match on Pinterest.

• robots.txt token: Pinterestbot
• IP list: none as a file. Pinterest gives one US range in prose, and says non-US addresses have no fixed range.
• Reverse DNS: names ending in pinterest.com or pinterestcrawler.com, confirmed forward.

Pinterest says its crawled content does not train its image-generation model, Pinterest Canvas, and says nothing about other models. On our sites in 19-26 October 2026 it made 7 requests on one day, 5 of them for robots.txt, from Amazon’s network, flagged by Cloudflare as “Aggregator”.

The feed fetcher of Protopage, a start-page service whose users add RSS feeds. Protopage publishes no crawler documentation.

User agent, as it reaches our logs:

Protopage/3.0 (http://www.protopage.com)

Match on Protopage.

• robots.txt token: none documented.
• IP list: none. Protopage has no crawler page at all.
• Reverse DNS: none.

On our sites in 19-26 October 2026 it made 261 requests, 260 of them for the WordPress site’s /feed/, from two OVH addresses. Cloudflare flagged all of them as “Feed Fetcher”. Protopage users have the feed on their start pages, and blocking it would cut them off.

Comscore's contextual crawler, which rates and categorises pages for ad partners. Obeys robots.txt and Crawl-delay; no IP list.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)

The token is proximic, the name of the company Comscore bought, even though the page is branded Comscore.

• robots.txt token: proximic
• IP list: none. Comscore says its addresses change, and offers custom authentication headers for allowlisting instead.
• Reverse DNS: none.

It runs at about one request a second by default and requests broken URLs when advertisers strip query strings. On our sites in 19-26 October 2026 it made 9 requests to the static site, from Amazon’s network. Cloudflare flagged 7. Our firewall blocked all 9 at the edge.

A name in bot lists. Alibaba documents no crawler for its Qwen models. Every request with this name on our sites was fake.

User agent, as the scanner sent it to our sites:

Mozilla/5.0 (compatible; Qwenbot/1.0; +https://qwen.alibaba.com/)

No Alibaba page documents QwenBot or any other crawler for Qwen. Treat the name as unconfirmed.

• robots.txt token: none documented.
• IP list: none.
• Reverse DNS: none.

On our sites in 19-26 October 2026, all 741 requests with this name came from the Google Cloud scanner.

Reflection AI's crawler for building and improving its models. Publishes an IP list and reverse DNS names, and asks for robots.txt rather than 403s.

User agent, as it reaches our logs (the same as Reflection’s example):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Reflectionbot/1.0; +https://reflection.ai/bot) Chrome/151.0.0.0 Safari/537.36

Reflection says this is the only user agent it crawls with. Match on Reflectionbot.

• robots.txt token: Reflectionbot
• IP list (JSON): ips.json
• Reverse DNS: names under reflection.ai, such as reflectionbot-*.reflection.ai.

Full card: what it does, what you gain or lose, what we saw

A desktop SEO crawler that anyone with a licence runs from their own computer. It obeys robots.txt by default, but the user can turn that off.

User agent, as it reached our logs:

Screaming Frog SEO Spider/24.3

The number is the program’s version. Users can switch the user agent to Googlebot, Bingbot, a browser or anything else. So the name appears only when someone leaves the default on.

• robots.txt token: Screaming Frog SEO Spider
• IP list: none. It crawls from the user’s own computer or network.
• Reverse DNS: none.

Screaming Frog puts responsibility for ignoring robots.txt on the user. On our sites in 19-26 October 2026 it made 2 requests to the WordPress site, robots.txt and one page, in the same second, from a single address.

The crawler of Seekport, a search engine run by the SEO-tool company SISTRIX. Obeys robots.txt and Crawl-delay.

User agent, as it reaches our logs (identical to Seekport’s page):

Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com)

Match on SeekportBot.

• robots.txt token: SeekportBot
• IP list: seekportbot_ips.txt, plain text with about 500 single addresses, not JSON.
• Reverse DNS: none published.

On our sites in 19-26 October 2026 it made 21 requests from Hetzner’s network; Cloudflare flagged 11 of them. Our firewall answered all 21 with a 403 at the edge.

Semrush's backlink crawler, one of about ten Semrush bots with separate tokens. Its Site Audit can crawl under an OpenAI-Search user agent.

User agents, as they reach our logs:

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)

• First string: the backlink crawler. ~bl marks backlinks.
• SemrushBot-BA and SemrushBot-SI: Backlink Audit and the On Page SEO Checker. They are separate bots with their own tokens.

Semrush’s pages give tokens, not full strings. Match on SemrushBot, and on SiteAuditBot and SplitSignalBot if you mean the whole family.

• robots.txt tokens: SemrushBot, plus SiteAuditBot, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SplitSignalBot, SemrushBot-OCOB, SemrushBot-FT, SemrushBot-ESI and RyteBot for the other tools.
• IP list: none for SemrushBot; Semrush says its addresses are not in blocks and not to block by IP. Site Audit uses 85.208.98.128/25 and SemrushBot-SI 85.208.98.0/24, given in Semrush’s help articles.
• Reverse DNS: none published.

Full card: what it does, what you gain or lose, what we saw

SentiOne's brand-monitoring crawler. It calls itself 'compatible with Googlebot' and follows your Googlebot rules when robots.txt has no group for it.

User agent, as it reaches our logs:

SentiBot www.sentibot.eu (compatible with Googlebot)

The string contains “Googlebot”, so a classifier that looks for Googlebot first counts every SentiBot request as a fake Googlebot. Match on SentiBot first.

• robots.txt token: sentibot
• IP list (JSON): sentibot-ips.json
• Reverse DNS: names ending in sentione.com, confirmed forward.

Its documentation lives on a Google Sites page that sentibot.eu redirects to. On our sites in 19-26 October 2026 it made 8 requests on 3 days, from Hetzner’s network, and Cloudflare flagged 2. Our firewall blocked 3, and the origin refused 3 more.

Serpstat's crawler for its backlink database. Obeys robots.txt and Crawl-delay, and publishes its addresses as a plain-text list.

User agent, in the form our logs saw:

serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; [email protected])

Serpstat’s page gives the token, not a full string. Match on serpstatbot.

• robots.txt token: serpstatbot
• IP list: serpstatbot-ip.txt, plain text with single addresses, not JSON.
• Reverse DNS: none published.

Serpstat reads robots.txt answers differently from Majestic. An unreachable file means “crawl allowed”, and a 403 means “no restrictions”. It keeps following old 404 and 301 URLs and crawls nofollow links. On our sites in 19-26 October 2026 the only 2 requests with this name came from a single test address, not from Serpstat.

Parallel Web Systems' crawler for the search index behind its web APIs for AI agents. Publishes an IP list.

User agent, as it reaches our logs (identical to Parallel’s docs):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0

No link and no contact in the string. Match on ShapBot; the user-triggered sibling is Shap-User/0.1.0.

• robots.txt token: ShapBot
• IP list (JSON): shapbot.json. Parallel also signs requests with Web Bot Auth.
• Reverse DNS: none published.

On our sites in 19-26 October 2026 it made 102 requests on 3 days, all from addresses on Parallel’s list (on Google Cloud) and all flagged by Cloudflare as “AI Search”. 60 got a redirect, and none asked for /robots.txt.

Slack's link previews and image proxy. Slack ignores robots.txt for them because they are not crawlers; changing previews means contacting Slack.

User agents, as they reach our logs:

Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Slack-ImgProxy (+https://api.slack.com/robots)
Slackbot 1.0 (+https://api.slack.com/robots)

• LinkExpanding: builds the preview when someone posts a link, reading only the start of the page through Range requests.
• ImgProxy: fetches and caches posted images; Slack documents it with a version number that our logs lacked.
• Slackbot 1.0: the generic agent for webhooks and integrations.

Match on Slack.

• robots.txt token: none; Slack ignores robots.txt.
• IP list: none.
• Reverse DNS: none.

Slack caches responses for about 30 minutes. On our sites in 19-26 October 2026 it made 73 requests, from Amazon networks, all to the WordPress site. Cloudflare flagged 51 of them, as “Page Preview” or “Webhooks”.

Trendiction's crawler for news, forums and blogs. The data is resold to market-research firms and apps. Its user agent mimics Firefox on Windows.

User agent, as it reaches our logs:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0

A Firefox-on-Windows string with the bot’s name wedged inside. Match on trendictionbot.

• robots.txt token: trendictionbot
• IP list: none.
• Reverse DNS: none.

Trendiction, from Luxembourg, crawls news sites, forums and blogs, comments included. It sells the filtered data to market-research firms, agencies and other apps. On our sites in 19-26 October 2026 it made 6 requests on 2 days, half for robots.txt, from Hetzner’s network, flagged by Cloudflare as “Monitoring & Analytics”.

X's crawler for link cards. X's documentation could not be read for this card; it reads robots.txt on almost every other visit.

User agent, as it reaches our logs:

Twitterbot/1.0

Match on Twitterbot.

• robots.txt token: Twitterbot
• IP list: not checked. X’s developer pages refused our tools, so this card needs a manual check.
• Reverse DNS: not checked.

On our sites in 19-26 October 2026 it made 70 requests on 7 days to the WordPress site, from X’s network (AS13414), all flagged by Cloudflare as “Page Preview”. 33 of them, almost half, were for robots.txt.

Yandex's main search crawler. Ignores Crawl-delay since 2018 and publishes no IP list; verify by reverse DNS.

User agent, as it reaches our logs:

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Yandex also documents a rendering variant with a Chrome version appended, and a MirrorDetector variant. Match on YandexBot.

• robots.txt token: YandexBot, case-sensitive.
• IP list: none. Yandex says its addresses change too often to publish.
• Reverse DNS: names ending in yandex.ru, yandex.net or yandex.com, confirmed forward.

On our sites in 19-26 October 2026 it made 18 requests flagged by Cloudflare, from Yandex’s network (AS13238). Our firewall answered all of them with a 403 at the edge.

You.com's search crawler. Obeys robots.txt and Crawl-delay, and can be verified by reverse DNS. Every request with its name on our sites was fake.

User agent as You.com documents it, then the one the scanner sent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36
Mozilla/5.0 (compatible; YouBot/1.0; +https://you.com/bot)

The real one is a Chrome string with env:prod and a docs.you.com link; the short form is the scanner’s. Match on YouBot.

• robots.txt token: YouBot
• IP list: none as a file. You.com gives one range on its page, 68.67.112.0/24, and publishes signing keys for Web Bot Auth.
• Reverse DNS: names like youbot-1-2-3-4.search.you.com, confirmed forward.

On our sites in 19-26 October 2026, all 775 requests with this name came from the Google Cloud scanner; You.com did not visit.

ZoomInfo's crawler for pages about companies and business people, for its B2B contact database. Obeys robots.txt, one request at a time.

User agent, as it reaches our logs:

ZoominfoBot (zoominfobot at zoominfo dot com)

Match on ZoominfoBot. Cloudflare files it under “Search Engine Crawler”, but what it builds is a sales-contact database.

• robots.txt token: Zoominfobot
• IP list: none.
• Reverse DNS: none.

ZoomInfo says it never opens more than one request at a time to a site, and may request links that do not exist because it misreads JavaScript. On our sites in 19-26 October 2026 it made 12 requests on one day to the WordPress site, 8 of them for robots.txt, from Google Cloud addresses.