GPTBot obeyed a robots.txt rule it never fetched. Meta read it and crawled anyway.

A month ago I published a finding that sounded like an accusation: GPTBot never fetched our robots.txt. In 139 days it made 805 requests and asked for the rules zero times, while a separate OpenAI bot, an OAI-SearchBot variant carrying a robots.txt; token in its user agent, fetched the file seconds before most of GPTBot’s crawl sessions.

The obvious follow-up question was the one I could not answer then: does it matter? A crawler that never reads the rules might ignore them. Or OpenAI might route the rules to it through another agent, so GPTBot obeys a file it never opened. The only way to tell the difference was to change the rules and watch.

So on 3 October I did. I changed the robots.txt on next.jsseo.dev to disallow every product cell for every user agent, 11 of the 55 test cells, and left the other 44 alone. Then I waited.

GPTBot obeyed. Meta did not, and not because it missed the change: it read the new rules, and minutes later crawled the pages they disallowed.

The experiment

next.jsseo.dev is a 55-cell test bed: the same content patterns served across five page types. Before 3 October the robots.txt said allow-all. The Disallow test changed one thing: every product cell, one fifth of the matrix, was made off-limits to all agents. The sitemap still listed them. The pages still linked to them. Only the rule changed.

The design is a natural control: the same bots keep visiting the same domain, so a bot that respects the rule keeps fetching the other 44 cells and stops fetching product. A bot that ignores the rule keeps the page-type mix it had before.

I left out the first 48 hours, because crawlers cache robots.txt (Google for up to 24 hours). The plan set a 4-week window, extended up to 8 weeks if the main crawlers had not each made two crawl sessions by then. Meta’s crawler had made none by 2 November, so the window kept running. This post uses 38.8 days, up to the snapshot on 13 November.

One rule for what counts as “the real bot”: only verified requests. IP-list or reverse-DNS checks for the operators that publish them, our ASN check for Meta. Spoofers ignore robots.txt by definition, so leaving them in would manufacture violations.

GPTBot obeyed a file it never read

GPTBot made 104 verified cell fetches in the window and touched a product cell zero times. If it had kept its baseline mix, it should have hit about 11 disallowed pages. The chance of seeing zero from that mix is well under one in ten thousand.

The session view is clearer than the statistic. Before the change, GPTBot’s routine crawl was 12 to 14 cells, and every one of those sessions included exactly one product cell. After the change the same routine came back five times as 13 cells, with the product cell missing. Its one large session went from 43-44 cells with 7 product cells to 39 cells with none. It crawled the same list with the disallowed URLs taken out.

And GPTBot still never asked for robots.txt itself: 0.0 requests per week before the change, 0.0 after. The file it obeyed is one it never opened.

What did read the rules for it? The tagged OAI-SearchBot. It read the new file 0.8 hours after the rule went live and kept reading it; GPTBot’s first crawl after the change came two weeks later.

ClaudeBot respected the rule too, across 13 sessions. CCBot’s zero is a single 24-cell session, which started in the same minute as a robots.txt read: consistent with compliance, but one session is not a test.

Meta read the rules, then crawled the disallowed pages

Two Meta crawlers fetched the disallowed cells.

meta-webindexer crawled the whole matrix twice, on 6 and 12 November: 57 cell fetches each time, 11 of them on product cells, as if the rule did not exist. Meta-ExternalAgent made one 16-cell session on 12 November, and 2 of those fetches were product cells.

The plan made no prediction for Meta, because neither of its crawlers has ever requested robots.txt on this domain. But Meta as a company does read it. facebookexternalhit, its link-preview fetcher, requested robots.txt 9 times after the change, from Meta’s network (AS32934), and both crawl sessions came right after one of those reads:

• 6 November: robots.txt read at 13:39 UTC, meta-webindexer starts its sweep at 13:40.
• 12 November: robots.txt read at 10:58 UTC, both crawlers fetch cells from 11:11 to 11:40.

This is the same split I found for OpenAI: one agent reads the rules, another crawls. The difference is what the crawler does with them. OpenAI’s crawler skipped the disallowed pages. Meta’s crawlers fetched them within minutes of the read.

The link between Meta’s reader and its crawlers is timing, not proof of a shared pipeline. What the logs do show is that Meta had the current rules in hand and its crawlers did not follow them.

Amazonbot stopped too, and I cannot see how

Amazonbot also shows zero: 91 cell fetches, almost every one its own single-page session, and no product cell. If it had kept its baseline share, about 7 should have landed on product. The chance of zero is about one in a thousand.

The puzzle is that no verified Amazon request has ever read robots.txt on this domain, before the change or after. Two explanations remain open:

• Amazon reads the rules somewhere we cannot verify. The one candidate in the logs is an agent on Amazon’s cloud network that reads robots.txt about every 18.5 hours, using the base string of Amazonbot’s older user agent with a (Gort) token. None of its 94 addresses is on any of Amazon’s published crawler IP lists, and they reverse-resolve to generic EC2 hostnames, so I cannot attribute it to Amazon.
• Amazonbot’s interest in product cells was fading anyway. Its product share by four-week block went 13.7%, 13.7%, 16.7%, 10.0%, 5.4%, then zero after the change. Even against the last 5.4%, a zero across 91 fetches is unlikely (about 0.6%), but the trend makes it less clean than GPTBot’s.

GPTBot shows that a crawler can obey a file it never fetches. Amazonbot shows the other side of that: when no reader is visible, a zero is a result without a mechanism.

Who read the new rules, and when

The switch gives a timestamp. For each operator I recorded the first robots.txt read after the change, by the crawler or by the operator’s reader, and counted product fetches after that read.

This window includes the 48-hour grace period, so Amazon’s count is 94 here and 91 in the test above. Meta’s row covers both of its crawlers.

A first re-read time is not a cache length. A cache only shows when a crawler fetches pages between the change and its next read, and here none of them did:

• OpenAI re-read within the hour; GPTBot did not crawl a cell until day 14.
• Google re-read at 18 hours and Googlebot fetched cells in the same moment.
• Microsoft re-read at 24 hours; Bingbot’s first cell fetch came on day 4.
• Anthropic re-read at 12 hours, but not because it cached longer: ClaudeBot’s two-hour robots.txt heartbeat had stopped 15 hours before the change and resumed 12 hours after it, then ran every two hours again. Its first cell fetch came on day 4.
• Apple re-read only after eight and a half days, but it did not crawl any cell until a month in, long after that read.
• Common Crawl re-read after 40 days, in the same minute as its only crawl session.

So this switch does not show any operator crawling on a stale copy. The question of how long each one would keep an old copy needs a short-lived rule, which is the next test.

What this means for site owners

• A robots.txt rule can be enforced even when the crawler you block never opens the file. GPTBot obeyed a Disallow it never fetched, because OpenAI reads the rules for it through another agent. If you filtered your logs for GPTBot next to robots.txt, you would see nothing and wrongly conclude OpenAI ignores you.
• Count rules-reads per operator, not per user agent. The tagged OAI-SearchBot robots.txt; fetch is OpenAI reading your rules, and facebookexternalhit reads them for Meta. Log analysis that splits on user agent alone misses both.
• Reading the rules is not obeying them. Meta had the current file minutes before it crawled the disallowed pages. If you need to keep content away from Meta’s crawlers, on this evidence robots.txt is not enough.
• A zero on your blocked paths needs a mechanism before you credit it. Amazonbot’s zero is real, but no verified Amazon agent read the rule. Check who reads your rules, not only who stays away.

What this does not prove

• One small domain. A large site with high crawl pressure may see different fetch patterns, and operators may split rules-reading differently there.
• We tested one kind of rule. We disallowed one page type by full paths. A rule against a whole directory, wildcards, or Disallow: / may be handled differently.
• Fetching is not indexing. We see requests, not indexes. We did not measure whether the disallowed cells left any index.
• Meta’s evidence is two sessions, both from the extended window. Before 2 November Meta did not crawl any cell. And the link between facebookexternalhit and Meta’s crawlers is timing: with no published IP list, both rest on Meta’s network (AS32934).
• This is an interim cut. Meta-ExternalAgent has one crawl session so far, so the plan allows the window to run to 30 November. More data can still turn a “respects” into an “ignores”, not the other way round.
• Some bots had too little traffic. Googlebot, Bingbot and Applebot fetched 0 to 6 cells in the test window, so their zeros mean “too little traffic to tell”, not compliance.

Method notes

• Window: the rule went live 2026-10-03 14:50 UTC; the test window runs from 48 hours later to 2026-11-13 10:24 UTC. Baseline: the 8 weeks before the change.
• Deduplication: the tracker logs each request twice, from Cloudflare’s feed and the app middleware. I count every Cloudflare row, plus a middleware row only when Cloudflare has no row for the same address, path and user agent within three seconds.
• Trust: a request counts only when its operator’s verification passes: published IP lists for OpenAI and Anthropic, IP list or reverse DNS for Amazon and Common Crawl, forward-confirmed reverse DNS for Google, Bing and Apple, AS32934 for Meta’s classes. Everything else is reported but does not decide a verdict.
• Decision rule: a bot “respects the Disallow” when it makes zero product fetches and the probability of that zero, given its baseline mix, is under 0.05 (one-sided binomial). One product fetch after the grace period means “ignores it”. Sessions split at 30 minutes of silence.
• Pre-registration: the plan, hypotheses and decision rule were committed to the project repository on 2026-10-03 at 14:34 UTC, 16 minutes before the rule went live (commit 9bb5c8d). The predictions for the cache switch were committed on 5 October (commit 1015fb0).
• Placebo: run on data from before any rule existed, with a fake start a month earlier, the same script found no bot that “respects the Disallow”.
• Analysis: scripts/analysis/robots_disallow.py and scripts/analysis/robots_cache.py switch, standard library only.

Data availability: methodology, tracker code and classifier are in the repository at github.com/Qbeczek1/jsseo-dev, private for now; access on request at [email protected].

Bias disclosure: I run JS SEO Lab as an independent technical SEO research project, and do paid technical SEO and AI-visibility audits through FratreSEO. No crawler vendor, search engine, or AI company funds this work.