Methodology

This is a cliff-notes version. The full METHODOLOGY.md in the repo is the canonical source.

What we measure

Pattern survival: which JS content patterns survive in bot HTTP responses, what LLMs report seeing when asked to fetch URLs containing those patterns, and how those signals differ across bot classes within a single framework.

Rendering mode (SSR / CSR / SSG) is a 2-3 mode control axis, not the primary variable. The primary axis is JS content pattern.

The 8 patterns

Each pattern is tested across 5 page types in SSR mode, plus baseline (clean) and CSR (negative control).

clean
No JS-injected content. All content in SSR HTML. Baseline.
js-images
<img src> attribute set by client JS after mount (useEffect setter pattern).
js-links
Navigation via onClick={navigate} instead of <a href>.
click-reveal
Main content body hidden until user clicks "show more".
js-fetched
Content fetched from API client-side after mount (price, reviews, recommendations).
hash-routing
URLs use fragment #/path instead of real paths.
late-loaded
Content rendered after IntersectionObserver fires.
mixed
Realistic combination of the above (e.g. product page with js-fetched price + lazy images + click-reveal reviews).

Pre-registered hypotheses

Committed to the repository before data collection started:

  1. H1 (sanity check). clean SSR ≡ clean SSG for batch crawlers. If this fails, methodology is broken.
  2. H2. clean SSR fully visible to all batch AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot).
  3. H3 (main test). Patterns js-images, js-links, click-reveal, js-fetched, hash-routing, late-loaded produce invisible main content to batch AI crawlers.
  4. H4. Googlebot (and Gemini via shared infrastructure) eventually renders all 8 patterns, with measurable per-pattern delay vs clean.
  5. H5. On-demand fetch tools (ChatGPT-User, Claude-User, Claude-SearchBot, Perplexity-User, Bing Copilot) - behaviour is exploratory; no strong prior. The split between batch crawler (GPTBot) and on-demand counterpart (ChatGPT-User) for the same provider is the most-watched signal.
  6. H6. Bing rendering of JS patterns is inconsistent - first systematic Bing data since the 2019 Edge crawler note.

Kill criterion: if pattern js-images shows no visibility difference vs clean for batch AI crawlers, the methodology is broken and investigation halts. Vercel/MERJ 2024 establishes a strong baseline that this prediction should hold.

Note, 2026-10-06: H1 to H6 are tested on the planned Phase 1 window, 2026-05-18 to 2026-08-07, in the Phase 1 retrospective (planned for December 2026). Data collected after 2026-08-07, and any result outside H1 to H6, is exploratory: reported with its own window, and a candidate for a new pre-registered test rather than evidence for or against H1 to H6. Experiments with their own plan written before they started, such as the robots.txt tests, are confirmatory for their own predictions.

Three-layer bot tracking

Every hit is captured by two request-observation layers, with a third layer for post-hydration JS execution:

  1. Layer 1: Next.js middleware. Every dynamic-route request hitting the test bed worker is logged before response is sent. Fire-and-forget POST to track.jsseo.dev/api/hit via ctx.waitUntil().
  2. Layer 2: Cloudflare GraphQL Analytics ingester. Every request that Cloudflare logs at the edge is polled into the tracker, including prerendered static routes that bypass the worker. 60s poll lag, dedup via natural-key hash.
  3. Layer 3: client-side JS beacon. Live dual-beacon setup. V1 is an inline script that sends an image-shaped request through /api/js-executed; V2 is a React client component that sends a fetch() POST through /api/js-executed-v2. The pair is run side by side because bot sandboxes block different network primitives. A beacon row means the renderer executed enough JavaScript to fire that beacon, not an absolute guarantee that every client-rendered content path completed. See the TRACKER.md for the current implementation status.

Bot identification

As of 2026-10-06, 41 user-agent classes have shown up in the tracker; 28 of them are named bots that visited the test cells. The terms are used the same way everywhere on this site:

user-agent class
One bucket from the tracker's classifier, human included.
named bot
A class that names a specific crawler or agent. Leaves out human and the classes that are not one crawler: unidentified bots (other-bot), scripts and headless browsers (programmatic, headless-chrome), cut-off user agents (truncated-ua), Cloudflare's own fetches (cf-internal) and scanners (security-scanner). A named bot is a user-agent claim; whether the claim is genuine is checked separately, below.
test cell
One of the 55 pages on next.jsseo.dev (pattern × page type × rendering mode).

How each bot claim is checked:

  • rDNS (Google, Bing, Apple, Baidu) - reverse DNS + forward confirmation against the bot's published domain (*.googlebot.com, *.search.msn.com, *.applebot.apple.com, *.baidu.com / *.baidu.jp).
  • IP-range (OpenAI, Perplexity, Anthropic, Amazon's Amzn-SearchBot and Amzn-User, Parallel's ShapBot) - CIDR membership check against published JSON manifests refreshed every 6 hours.
  • IP-range or rDNS (Amazonbot, CCBot) - a listed IP or a forward-confirmed hostname (*.crawl.amazonbot.amazon, *.crawl.commoncrawl.org) is enough; a hit is flagged only when both fail.
  • ASN (Meta) - Meta publishes neither an IP list nor an rDNS scheme, so its crawlers are checked against Meta's own network, AS32934, as reported by Cloudflare. That proves the request came from Meta, not which Meta service sent it.
  • claimed but not network-verified - Claude Code fetches come from the user's own network, so their UA claim stays visible but unresolved instead of being promoted to verified.
  • none / auto-unverified - deprecated anthropic-ai and Claude-Web UAs (pre-2024 vendor-deprecated), plus Google-Extended and Applebot-Extended, robots.txt tokens their vendors never send as a user agent, are flagged untrusted on insert, never reaching the verification queue.

HTTP Message Signature support (RFC 9421) is tracked as future verification work for signed agent traffic; unsigned crawler claims still rely on the paths above.

LLM fetch testing

Planned alongside bot crawl tracking: prompt-driven fetch tests across 9 LLM surfaces (ChatGPT, Claude.ai, Perplexity, Gemini, Bing Copilot - both web UI and API where available). Class A prompts ask the model to fetch a specific cell URL and describe what it sees. Each response is scored 0-4 (fetch failed / placeholder only / partial content / full but inaccurate / full and accurate), with binary flags for marker_detected, images_described, structured_data_used.

Planned cadence: weekly full sweep across all cells × all surfaces, daily focused probes on 24 high-signal cells, manual sampling 5-10 per week in chat UIs to catch divergence between API and chat surfaces.

None of this has run yet. So far there have only been one-off manual probes, such as the one behind NotebookLM ran our JavaScript. The first scored round is a manual test in October 2026: do these tools fetch a page that robots.txt disallows?

Descriptive, not inferential

N=1 per cell. Inferential statistics (p-values, confidence intervals) are misleading at this sample size. Reported metrics are descriptive: counts, rates, percentages, ratios. Claims are framed as descriptions of the test instance, not population-level inferences.

Open data, open code

Aggregated tracker data is public on the dashboard and in a dated snapshot; raw data is not published yet. Tracker, classifier and analysis scripts are in the repo. License: CC0 / CC-BY for data, MIT for code.

See /data for citation guidance and downloadable datasets.

Limitations

  • One framework (Next.js). Generalisation to Vue/Svelte/Solid/Qwik ecosystems requires Phase 2.
  • 8 patterns chosen as common, not exhaustive. Real-world sites use combinations and patterns not catalogued. The mixed cell explicitly tests realistic combos.
  • 2 search engines (Google, Bing). 9 LLM surfaces planned for fetch testing (large sample, not exhaustive).
  • 12 weeks of data collection, as planned for Phase 1 (2026-05-18 to 2026-08-07). Collection has continued since, and each finding states its own window. One geographic location for tracker server (Hetzner FSN1).

Results valid for the conditions tested. Extrapolation requires care.

Live data

/dashboard shows the current pattern survival matrix aggregated from the tracker. Updated daily during builds.

/test-bed explains the live research surface and links to specific cells for direct inspection.