Bytespider
ByteDance documents Bytespider as the crawler of Toutiao Search. The Bytespider strings most logs show are not the ones it documents.
| Operator | ByteDance |
|---|---|
| Type | AI: search index |
| robots.txt token | Bytespider |
| Follows robots.txt | Partly; Crawl-delay not documented |
| Sends visitors | Indirectly |
| Links to your pages | Not applicable |
| Can end up in AI training | Not documented |
| Verification |
reverse DNS
Reverse DNS under bytedance.com
Our tracker does not verify it yet. |
| Our call |
Publisher: No strong call E-commerce: No strong call Expert site: No strong call What the profiles mean |
How to recognize it
User agents as ByteDance documents them (desktop and Android; an iOS variant also exists):
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
What reached our logs instead:
Mozilla/5.0 (compatible; Bytespider; [email protected]) AppleWebKit/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])
The documented strings link to Toutiao’s webmaster site. The ones in our logs, and in most published bot lists, carry [email protected] or a document link instead. Match on Bytespider, then check the network.
- robots.txt token:
Bytespider - IP list: none as a file. ByteDance prints ten /24 ranges on its page, including
110.249.201.0/24,111.225.148.0/24and220.243.135.0/24. - Reverse DNS: names under
bytedance.com, such asbytespider-111-225-148-148.crawl.bytedance.com. ByteDance describes only the reverse lookup; confirm it forward anyway.
What it does
ByteDance’s webmaster documentation for Toutiao, its Chinese news and search app, names Bytespider as Toutiao Search’s crawler. It does not mention AI. Bot lists and blocklists usually call Bytespider ByteDance’s AI training crawler, but ByteDance’s own page does not say so, and we found no page by ByteDance about the spider-feedback variant.
What you gain by allowing it
- For a site with readers in China, presence in Toutiao Search. For a European site, very little that is documented.
What you lose
By allowing it
- Whatever the undocumented variant does with your pages, which nobody outside ByteDance can confirm.
By blocking it
- Toutiao Search, which matters only for Chinese audiences. Rate limits can also be set in Toutiao’s webmaster crawl-rate tool, which ByteDance says applies within a day.
How to verify it
Reverse DNS to bytedance.com (forward-confirm it), or a match against the ranges on ByteDance’s page. Our trackers do not verify Bytespider yet. Cloudflare did not flag any of the requests on our sites.
Gotchas and history
- Two Bytespiders: the documented search crawler and a widely logged variant with a feedback address are not described together anywhere official. Treat claims about either as unconfirmed.
- Corrected in our own post: Almost nobody in the West uses Kimi or Doubao first said ByteDance documents nothing checkable. The correction of 26 October 2026 adds the user agents, ranges and reverse DNS from its webmaster pages.
What we observed
Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).
- 730 requests carried a Bytespider user agent, and none came from ByteDance’s documented ranges:
- 691 came from the Google Cloud scanner;
- 39 came from Amazon and Zenlayer addresses with the
[email protected]mobile strings, and our firewall answered all 39 with a 403.
About ByteDance
ByteDance’s webmaster pages for Toutiao document Bytespider as Toutiao Search’s crawler. They give desktop, Android and iOS user agents, ten /24 ranges printed on the page, and reverse DNS under bytedance.com. The widely logged Bytespider strings with [email protected] are not on that page, and ByteDance documents no TikTok crawler.