CCBot
Common Crawl's crawler. Its open web archive is one of the most used sources of training data for large language models. Obeys robots.txt and Crawl-delay.
| Operator | Common Crawl |
|---|---|
| Type | AI: other |
| robots.txt token | CCBot |
| Follows robots.txt | Yes, by the operator's account; Crawl-delay honoured |
| Sends visitors | No |
| Links to your pages | No |
| Can end up in AI training | Yes |
| Verification |
reverse DNS, IP list
Reverse DNS under crawl.commoncrawl.org
Our tracker checks reverse DNS or the IP list. |
| Our call |
Publisher: Block E-commerce: Allow Expert site: Allow What the profiles mean |
How to recognize it
User agent, as Common Crawl documents it (the scanner sent the same string):
CCBot/2.0 (https://commoncrawl.org/faq/)
Match on CCBot; Common Crawl says the version number may rise. It also warns that other crawlers falsely claim to be CCBot.
- robots.txt token:
CCBot - IP list (JSON): ccbot.json
- Reverse DNS: names under
crawl.commoncrawl.org, IPv4 only, confirmed forward. Its IPv6 range has no reverse DNS, so use the list for those addresses.
What it does
CCBot builds Common Crawl’s open dataset, a free archive of the web that anyone can download for research and analysis. It checks robots.txt before fetching and obeys Crawl-delay. It honours nofollow and does not run JavaScript. Common Crawl also keeps a separate opt-out registry.
What you gain by allowing it
- Presence in the dataset that many language models learn from, far beyond one company. For an expert site that wants AI models to know its name and claims, this is one of the widest channels there is.
What you lose
By allowing it
- Your content in a public archive that anyone can use, for any model, under terms you do not negotiate.
By blocking it
- Presence in future Common Crawl releases, and with them in many models’ training data at once. A publisher protecting its content gets the broadest single opt-out; an expert site loses the broadest single channel.
How to verify it
Forward-confirmed reverse DNS to crawl.commoncrawl.org, or a match against ccbot.json. Our trackers accept either. Cloudflare does not flag it as a verified bot.
Gotchas and history
- WAF templates: some block CCBot as a “bad bot”. Check yours if you want to be in AI training data, because a block quietly undoes that.
- Fakes: with so many sites allowing it, its name is a favourite costume. On our sites, every request carrying it was fake.
What we observed
Data: Cloudflare Logpush for two of our sites, 19-26 October 2026 (8 days).
- 731 requests carried the CCBot user agent:
- 726 came from the Google Cloud scanner and the other 5 from a single consumer address;
- none came from Common Crawl.
- The real crawler did not visit either site in those 8 days. Common Crawl crawls in periodic batches, so a quiet week is not a block.
About Common Crawl
Common Crawl is a non-profit that publishes a free archive of the web. Its one crawler, CCBot, obeys robots.txt and Crawl-delay, honours nofollow and runs no JavaScript. The archive is open to anyone, which is why it ends up in so many language models’ training data. Common Crawl also keeps a separate opt-out registry.