Empirical observations from the JS SEO Lab experiment: pattern survival per bot class, batch vs on-demand pathway differences, Bing rendering at pattern granularity. Each post anchored in tracker data with raw signals linked.
Reflection AI's crawler visited our test bed 69 times between 18 September and 25 October. On 12 of its 13 visits to client-side-rendered pages, a second address fetched the page again and ran the Rea...
In August, Meta's AI crawler visited with a Chrome-based user agent and executed both our inline script and the React app, usually within a second of fetching the page. 83 beacons between 10 and 20 Au...
139 days of logs from one test domain. GPTBot made 805 requests and never asked for /robots.txt. In 135 of its 190 crawl sessions, a robots.txt request from an OAI-SearchBot variant that carries a rob...
Chinese AI assistants have few users in Europe and North America, yet their makers crawl Western sites, and Western companies build on their open-weight models. I found nothing published on what these...
Since April 2026 Anthropic's Help Center has linked a crawler IP list at claude.com/crawling/bots.json. I checked ClaudeBot against the wrong Anthropic page, concluded no crawler list existed, and rep...
In Cloudflare's logs, ClaudeBot spent 80% of 871 requests on /robots.txt and /sitemap.xml and 18% on content. The shape explains it: it swept the test bed in three short bursts (18-20 May, 3-5 July, 1...
Our JS-execution beacon fires through two channels - an inline script and a React client component. Over 80 days the bots sort into four tiers: renderers that evaluate React but skip inline scripts (A...
We flagged most of our Perplexity and ChatGPT crawler traffic as spoofed. Then we checked whether that was our mistake. It was not. All of the fake Perplexity, ChatGPT-User and OpenAI-search traffic t...
In 15 days of JS SEO Lab data, the Next.js test bed saw 6,125 hits across 25 user-agent classes. GPTBot, Meta-external, Amazonbot, and three Google sub-bots together produced 862 hits with zero reques...
In the first eight days of JS SEO Lab data, Bingbot fetched our test bed 126 times. 110 of those hits never executed JavaScript. The other 16 came from a different user-agent string, in six page loads...
We thought NotebookLM was scraping static HTML. A runtime-entropy probe proved otherwise: NotebookLM executed React, generated a UUID at runtime, fetched client-side content, and only appeared once we...
In the first week of JS SEO Lab data, OAI-SearchBot made 10 requests. Every one hit robots.txt. No homepage, no sitemap, no content cell. That does not prove how OpenAI crawls every site, but it does ...
Vercel and MERJ already settled that batch crawlers don't execute JavaScript. So why was I building a 6-mode renderability experiment? Here's why I rebuilt it as a pattern-axis instead - and what the ...