Correction, 2026-10-03: the first version said ClaudeBot crawled the content once and fetched no content after 2026-07-05. That was wrong. The counts came from the tracker’s Cloudflare-logs feed, which samples traffic and missed most of 14 July; the app’s own middleware log shows a third burst that day, 51 verified ClaudeBot content fetches across 39 cells between 13:32 and 14:58 UTC. The title, the timeline and the site-owner advice are corrected below. Counting the requests only the middleware saw, the split is 75% robots.txt and sitemap and 23% content, out of 926 requests. Two smaller fixes: the heartbeat ran at 12 + 12 a day on 11 of the 30 days, not every day, with a dip on 21-25 July; and the robots/sitemap pairs usually arrive within a second of each other, not always in the same second.

Update, 2026-09-25: the first version of this post said ClaudeBot has no public verification manifest. It has one: Anthropic publishes its crawler IP ranges at claude.com/crawling/bots.json, online since at least April 2026. The tracker now checks ClaudeBot against that list, and the spoofing section below uses the check. The findings do not change.

I expected crawlers to spend most of their requests on content. The point of a crawl is the content.

ClaudeBot spent 80% of its requests re-reading the rulebook and the map. The way it did it is the interesting part.

Crawl-budget allocation

Split every bot’s requests into four buckets - /robots.txt, /sitemap.xml, content cells, and everything else - over the full 80-day window (2026-05-18 to 2026-08-06, 15,874 canonical hits). The spread across the roster is enormous.

oai-searchbot claudebot gptbot meta-external amazonbot robots.txt sitemap content other
Figure 1 - Share of each bot's requests by target. Two poles: oai-searchbot and claudebot spend almost everything on robots.txt + sitemap; amazonbot and meta-external go almost entirely to content.
class total robots.txt sitemap content meta/content
oai-searchbot 210 189 1 6 31.7
claudebot 871 347 349 157 4.4
googlebot* 615 79 4 29 2.9
applebot 111 22 1 38 0.6
gptbot 298 0 79 171 0.5
seo-tools 601 160 7 419 0.4
bingbot 497 57 25 342 0.2
amazonbot 175 0 3 147 0.0
meta-external 102 0 1 75 0.0

meta/content = robots.txt + sitemap fetches per content fetch. *Googlebot’s total is inflated by a one-day spoof flood on / (a separate story); its real content sample is small.

Two opposite philosophies fall out of the last column. Compliance-heavy, content-light: oai-searchbot (90% robots.txt), claudebot (4.4 meta-fetches per page), googlebot. Content-greedy, rules-blind: amazonbot and meta-external fetched /robots.txt zero times in the entire window and went straight to content. ClaudeBot is the extreme of the first group: 696 combined robots + sitemap fetches against 157 content fetches.

The shape explains the number

The 18%-content figure hides a phase change. Split ClaudeBot’s daily activity into content vs meta (robots + sitemap) and a switch appears on 2026-07-05.

03060 last content fetch in Cloudflare logs: 2026-07-05 05-1805-2007-0407-0507-0707-1507-3108-05 contentrobots + sitemap
Figure 2 - ClaudeBot's daily content (blue) vs robots+sitemap (coral). Content stops after 2026-07-05 in Cloudflare's logs (a 14 July burst shows up only in the middleware log); the meta heartbeat runs at up to ~24/day for the next month, with a dip on 21-25 July.

In Cloudflare’s logs, ClaudeBot’s last content fetch was 2026-07-05. The middleware log adds one more burst on 2026-07-14: 51 verified content fetches across 39 cells in under 90 minutes, which Cloudflare’s sampled feed missed. Its last robots/sitemap fetch was 2026-08-06, still going at snapshot time. From 2026-07-07 it settled into 12 robots.txt + 12 sitemap.xml fetches a day: exactly that on 11 of the next 30 days, at least 11 of each on 23 of them, with a dip on 21-25 July.

The heartbeat is machine-regular

Zoom into one day (2026-07-15) and it is a metronome: robots.txt and sitemap.xml fetched as a pair, every ~2 hours, usually within a second of each other (81% of pairs within one second, 96% within five).

00:33  robots + sitemap      12:30  robots + sitemap
02:47  robots + sitemap      14:40  robots + sitemap
04:25  robots + sitemap      16:33  robots + sitemap
06:36  robots + sitemap      18:30  robots + sitemap
08:36  robots + sitemap      20:29  robots + sitemap
10:37  robots + sitemap      22:35  robots + sitemap

Twelve pairs, roughly two hours apart, no content in between. Not a crawl - a scheduled freshness poll.

This is real ClaudeBot, not a spoof

A User-Agent can be faked, so spoofing is a fair worry. Anthropic publishes the IP ranges its crawlers use, which makes it checkable: 834 of the 871 ClaudeBot hits (Cloudflare’s logs, 80-day window) come from Anthropic’s published range, and the 10 stragglers from cheap VPS ranges fail the check. They are the ~1% impersonation noise, not the heartbeat. The other 27 hits come from the same Amazon / AWS network as the verified ones but could not be checked, because an earlier version of the tracker had already discarded their IPs.

What this means for site owners

“ClaudeBot visits my pages regularly” is the wrong mental model. In this window it fetched the content in three short bursts and otherwise only polled robots.txt and sitemap.xml every two hours. The poll is not a content re-crawl.

If you want to control Amazonbot or Meta, robots.txt may be the wrong lever - they never fetch it. Zero robots.txt reads across the window for both. Whatever rule-checking they do is not happening on-domain, so a Disallow: they never read cannot be assumed to bind. The only way to know is to disallow and watch the content fetches, not the robots fetches.

What this does not prove

  • One cold-start domain, low popularity, 80 days. The cadence and the burst-then-heartbeat shape may be specific to how Anthropic treats a new, low-value property. A high-authority site with genuinely changing content could see a different allocation.
  • “Heartbeat ignores freshness” is not established. Content did not change during the heartbeat month, yet ClaudeBot re-fetched 39 cells once anyway, on 14 July. Whether bumping a page’s lastmod or body mid-heartbeat would pull ClaudeBot back to that page sooner is untested.
  • The 12/day count is unexplained. Why exactly two hours and twelve pairs - fixed interval, per-IP schedule summed across the pool, or coincidence of the window - is unknown.

What would falsify this

Mutate one content cell mid-heartbeat and watch the next two-hourly sitemap poll. If a content fetch of that cell follows, the poll is actionable and “ignores freshness” is wrong. If nothing follows for a week, the decoupling is real. I plan to run exactly that.

Method notes

Bucketing used exact url_path matching for /robots.txt and a LIKE '%sitemap%' match for the sitemap; content is any hit whose rendering_mode is ssr, csr, or ssg. The four-bucket totals per bot come from the canonical Cloudflare-logs ingest (deduplicated by ray_id); the Next.js middleware ingest is excluded to avoid double counting. Cloudflare’s feed samples traffic and can miss requests on busy days, as it did on 2026-07-14; the middleware log only sees page requests, never robots.txt or the sitemap. ASN attribution is from the Cloudflare edge log’s asn field.

Data availability: JS SEO Lab publishes methodology, tracker code, and classifier in the repository at github.com/Qbeczek1/jsseo-dev, private for now; access on request at [email protected]. Live dashboard at /dashboard/.

Bias disclosure: I run JS SEO Lab as an independent technical SEO research project. I also do paid technical SEO and AI-visibility audits through FratreSEO. No framework vendor, crawler vendor, search engine, or AI company funds this work.