All findings posts, chronological. For topical browsing, use topics or findings category.
In 15 days of JS SEO Lab data, the Next.js test bed saw 6,125 hits across 25 user-agent classes. GPTBot, Meta-external, Amazonbot, and three Google sub-bots together produced 862 hits with zero requests to /robots.txt. ClaudeBot read the rules seven times in 169 hits. OAI-SearchBot read them 38 times in 42. Three robots.txt behaviours, one rulebook, and a caveat for how to read your access logs.
In the first eight days of JS SEO Lab data, Bingbot fetched our test bed 126 times. 110 of those hits never executed JavaScript. The other 16 — using a different user-agent string — ran the full React pipeline. Naive bot trackers collapse both populations into one bingbot class. The split is invisible unless you parse the full UA string.
We thought NotebookLM was scraping static HTML. A runtime-entropy probe proved otherwise: NotebookLM executed React, generated a UUID at runtime, fetched client-side content, and only appeared once we changed the beacon from an image request to fetch().
In the first week of JS SEO Lab data, OAI-SearchBot made 10 requests. Every one hit robots.txt. No homepage, no sitemap, no content cell. That does not prove how OpenAI crawls every site, but it does raise a useful question about what OAI-SearchBot is actually doing on cold-start domains.
Vercel and MERJ already settled that batch crawlers don't execute JavaScript. So why was I building a 6-mode renderability experiment? Here's why I rebuilt it as a pattern-axis instead — and what the new test bed measures.