ClaudeBot crawled our content once, then pinged robots and sitemap every two hours for a month

I expected crawlers to spend most of their requests on content. The point of a crawl is the content.

ClaudeBot spent 80% of its requests re-reading the rulebook and the map. The way it did it is the interesting part.

Crawl-budget allocation

Split every bot’s requests into four buckets - /robots.txt, /sitemap.xml, content cells, and everything else - over the full 80-day window (2026-05-18 to 2026-08-06, 15,874 canonical hits). The spread across the roster is enormous.

meta/content = robots.txt + sitemap fetches per content fetch. *Googlebot’s total is inflated by a one-day spoof flood on / (a separate story); its real content sample is small.

Two opposite philosophies fall out of the last column. Compliance-heavy, content-light: oai-searchbot (90% robots.txt), claudebot (4.4 meta-fetches per page), googlebot. Content-greedy, rules-blind: amazonbot and meta-external fetched /robots.txt zero times in the entire window and went straight to content. ClaudeBot is the extreme of the first group: 696 combined robots + sitemap fetches against 157 content fetches.

The shape explains the number

The 18%-content figure hides a phase change. Split ClaudeBot’s daily activity into content vs meta (robots + sitemap) and a hard switch appears on 2026-07-05.

ClaudeBot’s last content fetch was 2026-07-05. Its last robots/sitemap fetch was 2026-08-06, still going at snapshot time. From 2026-07-07 onward it ran an almost perfectly regular 12 robots.txt + 12 sitemap.xml + 0 content, every single day for a month.

The heartbeat is machine-regular

Zoom into one day (2026-07-15) and it is a metronome: robots.txt and sitemap.xml fetched as a pair, in the same second, every ~2 hours.

00:33  robots + sitemap      12:30  robots + sitemap
02:47  robots + sitemap      14:40  robots + sitemap
04:25  robots + sitemap      16:33  robots + sitemap
06:36  robots + sitemap      18:30  robots + sitemap
08:36  robots + sitemap      20:29  robots + sitemap
10:37  robots + sitemap      22:35  robots + sitemap

Twelve pairs, roughly two hours apart, no content in between. Not a crawl - a scheduled freshness poll.

This is real ClaudeBot, not a spoof

ClaudeBot has no public verification manifest, so spoofing is a fair worry. It does not apply here: 861 of 871 ClaudeBot hits come from Amazon / AWS across 51 IPs. Anthropic runs on AWS, and a month of two-hourly clockwork is infrastructure, not a scraper. The 10 stragglers from cheap VPS ranges are the ~1% impersonation noise, not the heartbeat.

What this means for site owners

“ClaudeBot visits my pages regularly” is the wrong mental model. In this window it visited the content once, early, then stopped fetching pages entirely while continuing to hit robots.txt and sitemap.xml every two hours. The poll is not a content re-crawl.

If you want to control Amazonbot or Meta, robots.txt may be the wrong lever - they never fetch it. Zero robots.txt reads across the window for both. Whatever rule-checking they do is not happening on-domain, so a Disallow: they never read cannot be assumed to bind. The only way to know is to disallow and watch the content fetches, not the robots fetches.

What this does not prove

• One cold-start domain, low popularity, 80 days. The cadence and the front-load-then-heartbeat shape may be specific to how Anthropic treats a new, low-value property. A high-authority site with genuinely changing content could see a different allocation.
• “Heartbeat ignores freshness” is not established. Content did not change during the heartbeat month, so not re-crawling is the correct action. Whether bumping a page’s lastmod or body mid-heartbeat would pull ClaudeBot back to that page is untested.
• The 12/day count is unexplained. Why exactly two hours and twelve pairs - fixed interval, per-IP schedule summed across the pool, or coincidence of the window - is unknown.

What would falsify this

Mutate one content cell mid-heartbeat and watch the next two-hourly sitemap poll. If a content fetch of that cell follows, the poll is actionable and “ignores freshness” is wrong. If nothing follows for a week, the decoupling is real. I plan to run exactly that.

Method notes

Bucketing used exact url_path matching for /robots.txt and a LIKE '%sitemap%' match for the sitemap; content is any hit whose rendering_mode is ssr, csr, or ssg. The four-bucket totals per bot come from the canonical Cloudflare-logs ingest (deduplicated by ray_id); the Next.js middleware ingest is excluded to avoid double counting. ASN attribution is from the Cloudflare edge log’s asn field.

Data availability: JS SEO Lab publishes methodology, tracker code, and classifier in the public repository at github.com/Qbeczek1/jsseo-dev. Live dashboard at /dashboard/.

Bias disclosure: I run JS SEO Lab as an independent technical SEO research project. I also do paid technical SEO and AI-visibility audits through FratreSEO. No framework vendor, crawler vendor, search engine, or AI company funds this work.