Archive

ClaudeBot crawled our content once, then pinged robots and sitemap every two hours for a month

Across 871 requests, ClaudeBot spent 80% on /robots.txt and /sitemap.xml and 18% on content. The shape explains it: it swept the test bed once through 2026-07-05, then switched to a machine-regular heartbeat - robots and sitemap fetched as a pair every two hours, twelve times a day, zero content - and held it for a month. Amazonbot and Meta sit at the opposite pole: zero robots.txt, straight to content.

2026-08-06 by Jakub Sawa Findings

JavaScript execution has two depths, and bots split cleanly between them

Our JS-execution beacon fires through two channels - an inline script and a React client component. Over 80 days the bots sort into four tiers: renderers that evaluate React but skip inline scripts (Applebot, Googlebot-WRS, Read Aloud, NotebookLM), one that runs only the inline script (GPTBot), full executors (real browsers, Bing, GSC inspection), and a large zero-JS majority. Applebot rendered ~100% of what it crawled. Every batch AI crawler rendered nothing.

2026-08-06 by Jakub Sawa Findings

One IP claimed to be Google, OpenAI, Anthropic, Apple and Meta - while hunting for our .env file

We flagged most of our Perplexity and ChatGPT crawler traffic as spoofed. Then we checked whether that was our mistake. It was not. The entire apparent Perplexity, ChatGPT-User and OpenAI-search 'crawl' traces to nine cheap-VPS IPs, each rotating up to twelve vendor identities while scanning for secrets. Cloudflare 403s fake Googlebot but lets the fake AI-fetchers through as a 200 - so counting AI crawlers by user-agent overcounts Perplexity and ChatGPT roughly tenfold.

2026-08-06 by Jakub Sawa Findings

Most AI batch crawlers never fetched our robots.txt

In 15 days of JS SEO Lab data, the Next.js test bed saw 6,125 hits across 25 user-agent classes. GPTBot, Meta-external, Amazonbot, and three Google sub-bots together produced 862 hits with zero requests to /robots.txt. ClaudeBot read the rules seven times in 169 hits. OAI-SearchBot read them 38 times in 42. Three robots.txt behaviours, one rulebook, and a caveat for how to read your access logs.

2026-06-02 by Jakub Sawa Findings

Bingbot splits traffic between two subsystems. Only one of them executes JavaScript.

In the first eight days of JS SEO Lab data, Bingbot fetched our test bed 126 times. 110 of those hits never executed JavaScript. The other 16 — using a different user-agent string — ran the full React pipeline. Naive bot trackers collapse both populations into one bingbot class. The split is invisible unless you parse the full UA string.

2026-05-26 by Jakub Sawa Findings

NotebookLM ran our JavaScript

We thought NotebookLM was scraping static HTML. A runtime-entropy probe proved otherwise: NotebookLM executed React, generated a UUID at runtime, fetched client-side content, and only appeared once we changed the beacon from an image request to fetch().

2026-05-23 by Jakub Sawa Findings

OAI-SearchBot only fetched robots.txt

In the first week of JS SEO Lab data, OAI-SearchBot made 10 requests. Every one hit robots.txt. No homepage, no sitemap, no content cell. That does not prove how OpenAI crawls every site, but it does raise a useful question about what OAI-SearchBot is actually doing on cold-start domains.

2026-05-22 by Jakub Sawa Findings

Rendering mode was the wrong axis

Vercel and MERJ already settled that batch crawlers don't execute JavaScript. So why was I building a 6-mode renderability experiment? Here's why I rebuilt it as a pattern-axis instead — and what the new test bed measures.

2026-05-18 by Jakub Sawa Findings