Notes on how the experiment is conducted: pre-registration, verification (rDNS / IP-range / signature), data collection, statistical caveats.
139 days of logs from one test domain. GPTBot made 805 requests and never asked for /robots.txt. In 135 of its 190 crawl sessions, a robots.txt request from an OAI-SearchBot variant that carries a rob...
Chinese AI assistants have few users in Europe and North America, yet their makers crawl Western sites, and Western companies build on their open-weight models. I found nothing published on what these...
Since April 2026 Anthropic's Help Center has linked a crawler IP list at claude.com/crawling/bots.json. I checked ClaudeBot against the wrong Anthropic page, concluded no crawler list existed, and rep...
In Cloudflare's logs, ClaudeBot spent 80% of 871 requests on /robots.txt and /sitemap.xml and 18% on content. The shape explains it: it swept the test bed in three short bursts (18-20 May, 3-5 July, 1...
Our JS-execution beacon fires through two channels - an inline script and a React client component. Over 80 days the bots sort into four tiers: renderers that evaluate React but skip inline scripts (A...
We flagged most of our Perplexity and ChatGPT crawler traffic as spoofed. Then we checked whether that was our mistake. It was not. All of the fake Perplexity, ChatGPT-User and OpenAI-search traffic t...
In 15 days of JS SEO Lab data, the Next.js test bed saw 6,125 hits across 25 user-agent classes. GPTBot, Meta-external, Amazonbot, and three Google sub-bots together produced 862 hits with zero reques...
In the first eight days of JS SEO Lab data, Bingbot fetched our test bed 126 times. 110 of those hits never executed JavaScript. The other 16 came from a different user-agent string, in six page loads...
We thought NotebookLM was scraping static HTML. A runtime-entropy probe proved otherwise: NotebookLM executed React, generated a UUID at runtime, fetched client-side content, and only appeared once we...
In the first week of JS SEO Lab data, OAI-SearchBot made 10 requests. Every one hit robots.txt. No homepage, no sitemap, no content cell. That does not prove how OpenAI crawls every site, but it does ...
Vercel and MERJ already settled that batch crawlers don't execute JavaScript. So why was I building a 6-mode renderability experiment? Here's why I rebuilt it as a pattern-axis instead - and what the ...