How to recognize it

User agents, as they reach our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; ClaudeBot/1.0; [email protected])
  • First string: the plain crawler, which sent 97% of the verified requests.
  • Second string: a Chrome-based variant. It also fetches stylesheets, scripts and images, the way a browser loads a page.

Match on the token ClaudeBot. The old names anthropic-ai and Claude-Web are not in Anthropic’s current docs; any request using them is fake.

  • robots.txt token: ClaudeBot
  • IP list (JSON): bots.json, one list for all three of Anthropic’s named bots.
  • Reverse DNS: none published by Anthropic.

What it does

ClaudeBot collects web content that may be used to train Anthropic’s models. Anthropic says that blocking it excludes the site’s future content from training datasets. It honours robots.txt, including Crawl-delay, and does not try to get past CAPTCHAs.

What you gain by allowing it

  • Presence in what future Claude models know without searching. There is no link or click, and for an expert or brand site that presence is the point.

What you lose

By allowing it

  • Content used for training, with no traffic or payment in return.
  • Crawl load. It was the busiest verified crawler on our sites, ahead of Googlebot. Crawl-delay slows it down.

By blocking it

  • Nothing in Claude’s search or in fetches Claude makes for a user. Those are Claude-SearchBot and Claude-User, with their own tokens.
  • Presence in future training data. A publisher that licenses content keeps leverage; an expert site gives up a channel.

How to verify it

Match the IP against bots.json; the one-liner on the GPTBot card works with this URL. Cloudflare flags the real crawler as a verified bot in the category “AI Crawler”, and in our data the flag matched the list on every request. Anthropic recommends robots.txt rather than IP blocking, because its addresses can change.

Gotchas and history

  • Anthropic linked bots.json from its Help Center in April 2026, and an older Anthropic IP page lists different addresses; see Anthropic publishes its crawler IPs.
  • On a small site it can spend most of its requests on robots.txt and the sitemap between short content sweeps; see ClaudeBot crawled our content in three bursts.
  • Shared-hosting firewalls throttle it at the origin, out of sight of the CDN (see below).

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 4,937 requests carried a ClaudeBot user agent:
    • 4,205 (85%) came from Anthropic’s list, all from Amazon’s network (AS16509);
    • 710 of the 732 fakes came from the Google Cloud scanner;
    • Cloudflare’s verified-bot flag matched the list on all 4,937.
  • With 4,205 verified requests it was the busiest crawler on our sites; Googlebot made 2,502. 3,812 went to the static content site and 393 to the WordPress site.
  • 350 verified requests were for /robots.txt and 263 for sitemaps.
  • The Chrome-based variant made 125 requests. 56 of them were for stylesheets, scripts and images.
  • On the WordPress site, 44 of 393 verified requests got 403 or 429 from the origin’s hosting firewall. On the static site none did.

About Anthropic

  • ClaudeBot collects content that may be used to train Anthropic’s models. Blocking it excludes the site’s future content from training.
  • Claude-SearchBot indexes pages to improve Claude’s search results. Blocking it stops the site’s content from being indexed for that.
  • Claude-User fetches a page when a Claude user asks about it. Blocking it stops Claude from retrieving the site’s content for that user.
  • Claude Code is a developer tool. Its web fetches leave from the developer’s own machine, under a user agent that starts with Claude-User.

Anthropic says all three of its named bots honour robots.txt, including Crawl-delay, and do not try to get around CAPTCHAs. It recommends robots.txt over IP blocking.

One list, bots.json, covers all three named bots. Anthropic’s API sends web fetch, web search and MCP tool calls from a separate range, 160.79.104.0/21, documented on its API IP address page. That range is not in bots.json (see the Claude-User card).

Sources