Claude-SearchBot
Indexes pages to improve Claude's search results. On our sites every request with this name in October 2026 was fake.
| Operator | Anthropic |
|---|---|
| Type | AI: AI search |
| robots.txt token | Claude-SearchBot |
| Follows robots.txt | Yes, by the operator's account; Crawl-delay honoured |
| Sends visitors | Indirectly |
| Links to your pages | Sometimes |
| Can end up in AI training | Not documented |
| Verification |
IP list
Our tracker checks the IP list. |
| Our call |
Publisher: Allow E-commerce: Allow Expert site: Allow What the profiles mean |
How to recognize it
User agent, as the scanner sent it to our sites:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])
We have not seen a verified request, and Anthropic’s Help Center gives the name but not the full string. So treat this as the shape to match, not proof. Match on the token Claude-SearchBot.
- robots.txt token:
Claude-SearchBot - IP list (JSON): bots.json
- Reverse DNS: none published by Anthropic.
What it does
Claude-SearchBot crawls to improve the quality of Claude’s search results. Anthropic says that disabling it prevents your content from being indexed for search. It falls under Anthropic’s statement that its bots honour robots.txt and Crawl-delay.
What you gain by allowing it
- A place in Claude’s search results, where answers can cite and link your pages.
What you lose
By allowing it
- Crawl load, which we could not measure: the real bot did not visit.
- Anthropic’s page does not say whether this crawl also feeds training. Training is opted out separately, through ClaudeBot.
By blocking it
- Your pages are not indexed for Claude’s search. Claude-User can still fetch a page a user asks about.
How to verify it
Match the IP against bots.json; the one-liner on the GPTBot card works with this URL. On our sites no request with this name came from the list, and Cloudflare flagged none of them as verified.
Gotchas and history
- A dashboard counting it by user agent would show around 700 visits a week on our sites, and all of them are the scanner’s.
- Our test bed had not seen a real visit either when we checked it against the list; see Anthropic publishes its crawler IPs.
What we observed
Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).
- 734 requests carried a Claude-SearchBot user agent. None came from Anthropic’s list, and all 734 came from the Google Cloud scanner.
About Anthropic
- ClaudeBot collects content that may be used to train Anthropic’s models. Blocking it excludes the site’s future content from training.
- Claude-SearchBot indexes pages to improve Claude’s search results. Blocking it stops the site’s content from being indexed for that.
- Claude-User fetches a page when a Claude user asks about it. Blocking it stops Claude from retrieving the site’s content for that user.
- Claude Code is a developer tool. Its web fetches leave from the developer’s own machine, under a user agent that starts with
Claude-User.
Anthropic says all three of its named bots honour robots.txt, including Crawl-delay, and do not try to get around CAPTCHAs. It recommends robots.txt over IP blocking.
One list, bots.json, covers all three named bots. Anthropic’s API sends web fetch, web search and MCP tool calls from a separate range, 160.79.104.0/21, documented on its API IP address page. That range is not in bots.json (see the Claude-User card).