No AI crawler asked for our llms.txt. SEO tools did

Update, 2026-11-14: a day after this post went live, OpenAI’s search crawler asked for our llms.txt. Until this post, no page on the site linked to the file, and it was in neither the sitemap nor robots.txt. This post links to it. OAI-SearchBot fetched the post at 18:21 UTC on 13 November and /llms.txt at 14:19 UTC on 14 November, both from addresses on OpenAI’s published search-crawler list. It is the first verified request for the file from an OpenAI agent in our data. One request is not a pattern, and someone may have linked the file elsewhere at the same time. But it fits the simplest reading: crawlers treat llms.txt as an ordinary URL, ignored while nothing links to it and fetched once something does. A fetch is also not use; we still cannot see whether OpenAI does anything with the file. More on what each OpenAI agent fetches in GPTBot downloads your JavaScript. It does not run it.

jsseo.dev has served an llms.txt since the site went live in May 2026: a plain-text map of the site for language models, following the proposal at llmstxt.org. The idea is that AI systems read it to understand a site faster. I wanted to know which ones actually do.

In the 30 days to 26 October 2026, the answer was none. 31 requests from outside asked for the file, and not one came from a verified crawler of an AI company or a search engine. In the same 30 days, verified ClaudeBot asked for our robots.txt 241 times.

Three sites, one answer

I checked every place where we log bot traffic. Only jsseo.dev has the file; the others answer 404, which also tells you who looks for it.

My own checks are left out of all three counts. A request that went through a redirect counts once.

None of the 61 requests came from GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Googlebot, Bingbot, Applebot or any other crawler that passed verification. On jsseo.dev and the other two sites, Cloudflare did not mark any of them as a verified bot either; the test bed’s tracker does not record that flag.

The crawlers were there; they just did not ask

The big crawlers visited jsseo.dev in the same 30 days. They fetched pages, and most of them read robots.txt, often many times. None of them asked for llms.txt.

ChatGPT-User made 167 requests on behalf of people asking ChatGPT about the site. It never asked for the file written for exactly that situation. GPTBot never reads robots.txt itself, because OAI-SearchBot fetches it for OpenAI.

The other two sites, where the file does not exist, show the same pattern over eight days. Verified Googlebot read robots.txt 498 times there, ClaudeBot 356, OAI-SearchBot 133 and Bingbot 73. None of them asked for llms.txt even once, to see whether it was there.

Who does ask

The requests that come are from tools that check whether a site has the file. None of them is a model that would read it.

• SEOJuice-SearchBot sent 10 requests to jsseo.dev from six different networks, and 7 more to the other sites. SEOJuice is an SEO automation tool.
• Small crawlers, with their totals across our sites:
  • OraBot (7);
  • PipericBot (4);
  • UmaiBot (2);
  • DomainCrawler (2);
  • BitMoversBot (2);
  • PoweredByBot, from Keywords Everywhere (2), which asked for both llms.txt and llms-full.txt.
• BuiltWith, the technology profiler, sent 2. It ran from Google’s address space, so a log grouped by network makes it look like Google.
• mcpsee-observatory (mcpsee.com) asked for both files.
• Plain browser user agents sent 15 requests to jsseo.dev and 3 to the test bed. Almost all came from hosting and network providers, 12 of them from three Swedish ones. Real people rarely type /llms.txt; these look like audit scripts in browser clothing.
• PageSpeed Insights tested the test bed’s /llms.txt URL twice, so someone pasted it into Lighthouse.
• One Claude Code session, a developer’s coding assistant, fetched llms.txt on one of our sites. It was the only request from a major AI company’s product, and a person was driving it.

What to do with this

• Do not expect ChatGPT, Claude, Perplexity or Google to read your llms.txt. On our sites their crawlers read robots.txt and pages, and skipped it.
• If you keep one, know who reads it: SEO tools that score you for having it. It costs little to keep, but we found no evidence that it changes how AI systems see a site.
• Check your own logs before believing either side. Filter for /llms.txt, then verify the requester’s IP against the operator’s published list or reverse DNS, as each card in our bot catalog explains. A user agent alone proves nothing.

What this does not prove

• These are small sites. A large, well-linked site may see different behaviour. If major crawlers fetch llms.txt anywhere, they did not do it here.
• The window is short. Thirty days on jsseo.dev and eight on the other two sites. The test bed has five months, but it never served the file.
• The test bed half of the plan did not happen. Our methodology pre-registered an llms.txt on half of the test bed’s rendering modes. That was never set up, so the test bed only shows who looks for a missing file.
• We see requests, not use. An AI company could fetch llms.txt from a third-party copy, or use it when a user pastes the URL into a chat. Neither shows up in our logs.
• Cloudflare’s analytics for jsseo.dev are sampled at high volume. At these volumes the counts should be complete, but treat them as close rather than exact. The other two sites come from Logpush, which logs every request.

Method notes

• jsseo.dev: Cloudflare’s GraphQL analytics, 30 days, the most the dataset allows on our plan. Verification ran our tracker’s own code on each requesting address: the operator’s IP list, or reverse DNS with a forward check. Meta’s crawlers could not be checked this way, because the network field is not available in this query.
• next.jsseo.dev: the JS SEO Lab tracker, 18 May to 26 October 2026, 44,365 rows.
• The other two sites: Cloudflare Logpush into our bot-monitor, 19 to 26 October 2026.
• Exclusions: requests from the author’s own network are left out everywhere, 6 on jsseo.dev and 5 on the test bed.
• Paths counted: /llms.txt and /llms-full.txt.

Data availability: methodology, tracker code and classifier are in the repository at github.com/Qbeczek1/jsseo-dev, private for now; access on request at [email protected].

Bias disclosure: I run JS SEO Lab as an independent technical SEO research project, and do paid technical SEO and AI-visibility audits through FratreSEO. No crawler vendor, search engine, or AI company funds this work.