Amazonbot
Amazon's crawler for improving its products and services, which may include training its AI models. Honours robots.txt and noarchive, ignores Crawl-delay.
| Operator | Amazon |
|---|---|
| Type | AI: AI training |
| robots.txt token | Amazonbot |
| Follows robots.txt | Yes, by the operator's account; Crawl-delay not honoured |
| Sends visitors | No |
| Links to your pages | No |
| Can end up in AI training | Yes |
| Verification |
IP list
Our tracker checks reverse DNS or the IP list. |
| Our call |
Publisher: Block E-commerce: Allow Expert site: Allow What the profiles mean |
How to recognize it
User agent, as it reaches our logs:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
Match on Amazonbot. Amazon’s two other agents, Amzn-SearchBot and Amzn-User, use the same pattern with their own names.
- robots.txt token:
Amazonbot - IP list: ip-addresses, an HTML page, not a JSON file.
- Reverse DNS: not documented by Amazon. In practice, names under
crawl.amazonbot.amazonforward-confirm, and our trackers accept either the list or that check.
What it does
Amazon says Amazonbot is used to improve its products and services and may be used to train Amazon’s AI models. It respects robots.txt but not Crawl-delay. Amazon also honours three page-level robots meta tags:
noarchive: do not use the page for model training;noindex: do not index the page;none: do not index the page.
What you gain by allowing it
- Presence in what Amazon’s models and services learn from. There is no link or click.
What you lose
By allowing it
- Content used for training, unless pages carry
noarchive. - Crawl load that
Crawl-delaycannot slow down.
By blocking it
- Nothing in Alexa’s search, which uses Amzn-SearchBot. Amazon says that bot does not crawl for model training.
- Presence in Amazon’s training data.
noarchiveis a finer alternative: the page stays crawlable but is excluded from training.
How to verify it
Match the IP against Amazon’s list. The page is HTML, so a script has to parse the addresses out of it. Alternatively, reverse DNS to a name under crawl.amazonbot.amazon, confirmed forward. Cloudflare flags the real crawler as a verified bot in the category “AI Crawler”, and in our data the flag matched our check on every request.
Gotchas and history
- The
noarchivetag: Amazon reads it as “do not use for training”, while Bing reads it as “keep out of Copilot answers”. One tag can change your standing with two companies. - Propagation: Amazon says changes can take about 24 hours to reach its systems.
- No robots.txt reads: on our test bed in May it never requested robots.txt; see Most AI batch crawlers never fetched our robots.txt. Here it did, 4 times in 8 days.
What we observed
Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).
- 1,160 requests carried an Amazonbot user agent:
- 411 (35%) came from Amazon’s list, all from Amazon’s network (AS14618);
- 746 of the 749 others came from the Google Cloud scanner;
- Cloudflare’s flag matched our check on all 1,160.
- 409 of the 411 went to the static content site, and 238 of those (58%) got a 301 on 119 old addresses. Amazonbot keeps requesting URLs that redirected long ago.
About Amazon
- Amazonbot improves Amazon’s products and services and may train its AI models.
- Amzn-SearchBot feeds search in Amazon products such as Alexa. Amazon says it does not crawl for model training. Without a group of its own in robots.txt, it follows the rules given to other search bots.
- Amzn-User fetches pages for user actions, such as Alexa answers that need current information. It may not follow every robots.txt rule.
None of the three supports Crawl-delay. All of them honour the page tags noarchive (no training), noindex and none. Amazon publishes a separate IP list for each, as HTML pages rather than JSON files.