How to recognize it

User agent, as it reaches our logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)

Match on the token GPTBot; the version number changes. Then verify the IP: four out of five requests with this user agent on our sites were fake.

  • robots.txt token: GPTBot
  • IP list (JSON): gptbot.json
  • Reverse DNS: none published by OpenAI.

The lists for OpenAI’s other agents (OAI-SearchBot, ChatGPT-User, OAI-AdsBot) are on the OpenAI card.

What it does

GPTBot collects pages that OpenAI may use to train its generative AI foundation models. OpenAI says that disallowing GPTBot tells it the site’s content should not be used for that training.

It is one of four OpenAI agents, each with its own user agent, and OpenAI states that each robots.txt setting is independent of the others. Search results in ChatGPT come from OAI-SearchBot, and pages a user asks ChatGPT to open come from ChatGPT-User.

What you gain by allowing it

  • Presence in what future OpenAI models know without a search step: your brand, your terms, your claims. There is no link, click or attribution. For an expert site that wants to be the name a model mentions, this is the point.
  • No double load if you also allow OAI-SearchBot. OpenAI says it may use one crawl for both uses to avoid duplicate crawling.

What you lose

By allowing it

  • Your content trains models with no traffic, link or payment in return.
  • OpenAI’s crawler page does not say whether a later Disallow removes content collected earlier.

By blocking it

  • Nothing in ChatGPT search or in user-requested fetches, since those have their own tokens.
  • Presence in future training data. A publisher that wants to license its content keeps leverage by blocking. An expert or brand site gives up a channel.

How to verify it

Match the IP against gptbot.json:

curl -s https://openai.com/gptbot.json | python3 -c "import json,sys,ipaddress; ip=ipaddress.ip_address(sys.argv[1]); print(any(ip in ipaddress.ip_network(p.get('ipv4Prefix') or p['ipv6Prefix']) for p in json.load(sys.stdin)['prefixes']))" 203.0.113.7

Replace 203.0.113.7 with the IP from your log. Cloudflare marks the real GPTBot as a verified bot in the category “AI Crawler”; in our data that flag matched the IP list on every request.

Gotchas and history

  • Blocking GPTBot does not take you out of ChatGPT. For that, disallow OAI-SearchBot, and accept that ChatGPT-User may still fetch pages, because OpenAI says robots.txt may not apply to user-initiated actions.
  • Do not wait for a robots.txt request from GPTBot to check that it reads your rules. In our logs it never sent one. OpenAI says its robots.txt fetches may carry an extra robots.txt marker in the user agent. On our sites that request arrives as OAI-SearchBot (compatible; OAI-SearchBot/1.4; robots.txt; ...). The longer analysis is in GPTBot never fetched our robots.txt.
  • Firewalls below the CDN. Shared-hosting firewalls (ModSecurity rule sets) can throttle or block AI crawlers at the origin, where the CDN dashboard shows nothing blocked.
  • OpenAI gives about 24 hours for robots.txt changes to take effect for search. It gives no figure for GPTBot.

What we observed

Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).

  • 943 requests carried a GPTBot user agent:

    • 185 (20%) came from OpenAI’s published ranges;
    • 758 (80%) did not, and 749 of those came from the Google Cloud scanner;
    • Cloudflare’s verified-bot flag matched the IP list on all 943.

    Fake GPTBots outnumber the real one four to one, so any GPTBot count from user agents alone is mostly noise.

  • All 185 verified requests used GPTBot/1.4.

  • Verified GPTBot did not request /robots.txt once in 8 days.

    • Every verified robots.txt request from an OpenAI agent came from OAI-SearchBot, 132 of them.
    • 53 of those carried the robots.txt marker.
    • This matches 139 days on our test bed, where a marked OAI-SearchBot request arrived seconds before most GPTBot sessions.

  • On one site, 40 of the 185 verified requests got 403, 406 or 429 from the origin server, not from Cloudflare.

About OpenAI

  • GPTBot collects content for training foundation models. Disallowing it opts the site out of training only.
  • OAI-SearchBot finds pages for search results in ChatGPT. Disallowing it removes the site from those results. OpenAI gives about 24 hours for a robots.txt change to take effect there.
  • ChatGPT-User opens pages when a user asks ChatGPT or a custom GPT to. It does not crawl on its own, and OpenAI says robots.txt rules may not apply to it.
  • OAI-AdsBot visits only pages submitted as ads, to check the landing page.

OpenAI states that each setting is independent. When a site allows both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both. The common mix-up is blocking GPTBot to stay out of ChatGPT: that only stops training, while search visibility depends on OAI-SearchBot.

Sources