Baiduspider
Baidu's search crawler. One token covers all of Baidu's agents; it ignores Crawl-delay, and Baidu publishes no IP list on purpose.
| Operator | Baidu |
|---|---|
| Type | Search engine: search index |
| robots.txt token | Baiduspider |
| Follows robots.txt | Yes, by the operator's account; Crawl-delay not honoured |
| Sends visitors | Yes |
| Links to your pages | Yes |
| Can end up in AI training | Not documented |
| Verification |
reverse DNS
Reverse DNS under baidu.com, baidu.jp
Our tracker checks reverse DNS. |
| Our call |
Publisher: No strong call E-commerce: No strong call Expert site: No strong call What the profiles mean |
How to recognize it
User agents, as they reach our logs (desktop, then mobile):
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Baidu’s own pages print these with slightly different spacing, and one has a full-width bracket. Match on Baiduspider, after the more specific Baiduspider-render. Baidu also runs -image, -video, -news, -favo, -cpro and -ads variants.
- robots.txt token:
Baiduspider - IP list: none. Baidu says its ranges change and it cannot publish them.
- Reverse DNS: names under
baidu.comorbaidu.jp, such asbaiduspider-111-206-198-69.crawl.baidu.com, confirmed forward.
What it does
Baiduspider crawls the web to build Baidu’s search index, the largest in China. Baidu says it strictly follows robots.txt. It no longer reads Crawl-delay; crawl pressure is set in Baidu’s webmaster tool instead.
What you gain by allowing it
- Baidu search traffic, which matters only for an audience in China.
What you lose
By allowing it
- Crawl load that
Crawl-delaycannot reduce.
By blocking it
- Baidu search. A
Baiduspiderblock covers all Baidu agents, except the two ads crawlers that ignore robots.txt.
How to verify it
Reverse DNS to baidu.com or baidu.jp, then forward-confirm. Our trackers do this. Cloudflare flags the real crawler as “Search Engine Crawler” and matched our check on every request.
Gotchas and history
- Country rules: a country-based firewall rule for China blocks it, and with it Baidu’s ability to read your robots.txt.
- Ads crawlers: Baiduspider-cpro and Baiduspider-ads ignore robots.txt by Baidu’s own account.
What we observed
Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).
- 1,068 requests carried a Baiduspider user agent:
- 372 passed forward-confirmed reverse DNS, from Chinese networks (AS4837, AS23724);
- 694 came from the Google Cloud scanner.
- Our firewall answered all 372 real requests with a 403 at the edge, so Baidu never reached our robots.txt either. Neither site targets China.
About Baidu
- Baiduspider crawls for Baidu search, with product variants for images, video and news.
- Baiduspider-render loads pages with their CSS, JavaScript and images.
A Baiduspider group in robots.txt covers them all, except the ads crawlers Baiduspider-cpro and Baiduspider-ads, which Baidu says ignore robots.txt. Verification is by reverse DNS under baidu.com or baidu.jp. Baidu publishes no IP list and ignores Crawl-delay.