Author: Jakub Sawa

Reflectionbot runs JavaScript only when the HTML is empty

Reflection AI's crawler visited our test bed 69 times between 18 September and 25 October. On 12 of its 13 visits to client-side-rendered pages, a second address fetched the page again and ran the React app within three seconds. On 56 visits to server-rendered and static pages, including pages whose main content needs JavaScript, it fetched the HTML once and ran nothing. Every address it fetched pages from is on Reflection's published IP list and resolves to reflection.ai.

2026-10-25 Findings

Meta-ExternalAgent ran our JavaScript on 23 of 32 page fetches

In August, Meta's AI crawler visited with a Chrome-based user agent and executed both our inline script and the React app, usually within a second of fetching the page. 83 beacons between 10 and 20 August, every one of the 21 pages it fetched rendered at least once, all of it from Meta's network. Its earlier visits under the plain user agent came before our beacon existed, so we cannot say whether that version ran JavaScript.

2026-10-06 Findings

GPTBot never fetched our robots.txt. Another OpenAI bot did, seconds before most of its crawls

139 days of logs from one test domain. GPTBot made 805 requests and never asked for /robots.txt. In 135 of its 190 crawl sessions, a robots.txt request from an OAI-SearchBot variant that carries a robots.txt token in its user agent arrived within five seconds before GPTBot started. Bing and Meta show the same split more weakly. Googlebot, Applebot and ClaudeBot read their own rules. Amazonbot never asked at all.

2026-10-03 Findings

Almost nobody in the West uses Kimi or Doubao. I'm tracking their crawlers anyway

Chinese AI assistants have few users in Europe and North America, yet their makers crawl Western sites, and Western companies build on their open-weight models. I found nothing published on what these crawlers see on a JavaScript site, so I split 19 China-based agents out of the tracker's catch-all bucket. Four of the ten companies document their agents in a checkable way, five agent names have no real traffic behind them that I could find, and Manus-User passes for a human visitor. Five hypotheses and a four-week plan.

2026-09-26 Findings

Anthropic publishes its crawler IPs. I missed it for four months

Since April 2026 Anthropic's Help Center has linked a crawler IP list at claude.com/crawling/bots.json. I checked ClaudeBot against the wrong Anthropic page, concluded no crawler list existed, and repeated that for four months. Against the right list, 834 of 871 ClaudeBot requests check out, the 10 that fail come from known impersonators, and Claude-User's robots.txt check comes from addresses the list has never contained.

2026-09-25 Findings

ClaudeBot crawled our content in three bursts and pinged robots and sitemap every two hours in between

In Cloudflare's logs, ClaudeBot spent 80% of 871 requests on /robots.txt and /sitemap.xml and 18% on content. The shape explains it: it swept the test bed in three short bursts (18-20 May, 3-5 July, 14 July) and between them ran a machine-regular heartbeat - robots and sitemap fetched together every two hours, up to twelve times a day. Amazonbot and Meta sit at the opposite pole: zero robots.txt, straight to content.

2026-08-06 Findings

JavaScript execution has two depths, and bots split cleanly between them

Our JS-execution beacon fires through two channels - an inline script and a React client component. Over 80 days the bots sort into four tiers: renderers that evaluate React but skip inline scripts (Applebot, Googlebot-WRS, Read Aloud, NotebookLM), one that runs only the inline script (GPTBot), full executors (real browsers, Bing, GSC inspection), and a large zero-JS majority. Applebot rendered about 90% of what it crawled. No other named batch AI crawler rendered anything.

2026-08-06 Findings

One IP claimed to be Google, OpenAI, Anthropic, Apple and Meta - while hunting for our .env file

We flagged most of our Perplexity and ChatGPT crawler traffic as spoofed. Then we checked whether that was our mistake. It was not. All of the fake Perplexity, ChatGPT-User and OpenAI-search traffic traces to nine IPs, rotating up to thirteen vendor identities, most of them while scanning for secrets. Cloudflare 403s fake Googlebot but lets the fake AI-fetchers through - so counting AI crawlers by user-agent overcounts Perplexity and ChatGPT roughly tenfold.

2026-08-06 Findings

Most AI batch crawlers never fetched our robots.txt

In 15 days of JS SEO Lab data, the Next.js test bed saw 6,125 hits across 25 user-agent classes. GPTBot, Meta-external, Amazonbot, and three Google sub-bots together produced 862 hits with zero requests to /robots.txt. ClaudeBot read the rules seven times in 169 hits. OAI-SearchBot read them 38 times in 42. Three robots.txt behaviours, one rulebook, and a caveat for how to read your access logs.

2026-06-02 Findings

Bingbot splits traffic between two subsystems. Only one of them executes JavaScript.

In the first eight days of JS SEO Lab data, Bingbot fetched our test bed 126 times. 110 of those hits never executed JavaScript. The other 16 came from a different user-agent string, in six page loads, and five of those loads executed JavaScript. Naive bot trackers collapse both populations into one bingbot class. The split is invisible unless you parse the full UA string.

2026-05-26 Findings

NotebookLM ran our JavaScript

We thought NotebookLM was scraping static HTML. A runtime-entropy probe proved otherwise: NotebookLM executed React, generated a UUID at runtime, fetched client-side content, and only appeared once we changed the beacon from an image request to fetch().

2026-05-23 Findings

OAI-SearchBot only fetched robots.txt

In the first week of JS SEO Lab data, OAI-SearchBot made 10 requests. Every one hit robots.txt. No homepage, no sitemap, no content cell. That does not prove how OpenAI crawls every site, but it does raise a useful question about what OAI-SearchBot is actually doing on cold-start domains.

2026-05-22 Findings

Rendering mode was the wrong axis

Vercel and MERJ already settled that batch crawlers don't execute JavaScript. So why was I building a 6-mode renderability experiment? Here's why I rebuilt it as a pattern-axis instead - and what the new test bed measures.

2026-05-18 Findings