How to recognize it

User agent, as it reaches our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected])

Match on the token Claude-User, but check for claude-code/ first. Claude Code’s fetches use Claude-User (claude-code/...) and are a different agent.

  • robots.txt token: Claude-User
  • IP list (JSON): bots.json
  • Also from Anthropic: API tools (web fetch, web search, MCP) call out from 160.79.104.0/21, a range published on Anthropic’s API IP addresses page but not in bots.json.
  • Reverse DNS: none published by Anthropic.

What it does

Claude-User retrieves a page when someone asks Claude a question that needs it. Anthropic says that disabling it prevents Claude from retrieving your content in response to a user query. It falls under Anthropic’s statement that its bots honour robots.txt and Crawl-delay.

What you gain by allowing it

  • Claude can read your page when a user asks about it, and its answer may cite the page.

What you lose

By allowing it

  • Your content is summarised for a user who may not click through.
  • Anthropic’s page does not say whether these fetches are used for training.

By blocking it

  • Claude answers questions about your pages without reading them.
  • Training and Claude’s search index are unaffected; those are ClaudeBot and Claude-SearchBot.

How to verify it

Match the IP against bots.json; the one-liner on the GPTBot card works with this URL. Treat an address in 160.79.104.0/21 as Anthropic too. A strict allowlist built from bots.json alone blocks Claude-User requests that Claude’s API makes (see below).

Cloudflare flags the listed requests as a verified bot in the category “AI Assistant”.

Gotchas and history

  • Its robots.txt check does not always come from listed addresses. In May it came from Google Cloud addresses outside the list; see Anthropic publishes its crawler IPs. In October some of it came from Anthropic’s API range.
  • The listed Claude-User addresses are on Google Cloud (AS396982), the same network the scanner rents from. So an ASN rule cannot separate them; only the list can.

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 769 requests carried a Claude-User user agent:
    • 31 (4%) came from Anthropic’s list, all from Google Cloud addresses (AS396982);
    • 720 of the 738 failures came from the Google Cloud scanner;
    • Cloudflare’s verified-bot flag matched the list on every listed request.
  • 15 of the failures were not fakes. They came from 160.79.106.x, inside Anthropic’s API outbound range:
    • they arrived on one site between 22 and 26 October;
    • 5 were for /robots.txt, each sent one second before a page request from the same range;
    • our trackers first counted them as failed, because bots.json does not list the range. They now accept 160.79.104.0/21 for Claude-User, but not for ClaudeBot or Claude-SearchBot.
  • The 31 listed requests fetched 28 different pages, and 4 asked for /robots.txt.

About Anthropic

  • ClaudeBot collects content that may be used to train Anthropic’s models. Blocking it excludes the site’s future content from training.
  • Claude-SearchBot indexes pages to improve Claude’s search results. Blocking it stops the site’s content from being indexed for that.
  • Claude-User fetches a page when a Claude user asks about it. Blocking it stops Claude from retrieving the site’s content for that user.
  • Claude Code is a developer tool. Its web fetches leave from the developer’s own machine, under a user agent that starts with Claude-User.

Anthropic says all three of its named bots honour robots.txt, including Crawl-delay, and do not try to get around CAPTCHAs. It recommends robots.txt over IP blocking.

One list, bots.json, covers all three named bots. Anthropic’s API sends web fetch, web search and MCP tool calls from a separate range, 160.79.104.0/21, documented on its API IP address page. That range is not in bots.json (see the Claude-User card).

Sources