What CCBot Collects for Common Crawl's Open Web Archive

What is CCBot?

Common Crawl operates CCBot as the crawler that builds its free, open archive of the web, a project Common Crawl states has run continuously since 2007. The bot requests pages the same way any crawler does, and Common Crawl's own site reports the resulting corpus now holds over 300 billion pages, with 3 to 5 billion new pages added each month, checked 9 September 2026. That archive is public, so anyone, including a competing AI lab, a university researcher or RedReplier, can download the same pages CCBot collected.

What is CCBot's user agent string?

CCBot identifies itself with the exact string CCBot/2.0 (https://commoncrawl.org/faq/), according to Common Crawl's own CCBot documentation, checked 9 September 2026. Common Crawl warns on that same page that other crawlers falsely identify themselves as CCBot, so the string alone is a claim a request makes about itself rather than proof of origin. Match the source IP address against Common Crawl's published range before trusting the header.

Detail	Value	Source
User agent string	CCBot/2.0 (https://commoncrawl.org/faq/)	commoncrawl.org/ccbot
IP verification	Published as JSON at index.commoncrawl.org/ccbot.json	commoncrawl.org/ccbot
robots.txt token	CCBot	commoncrawl.org/ccbot
Corpus size	Over 300 billion pages since 2007	commoncrawl.org

What does Common Crawl publish?

Common Crawl publishes raw web page captures alongside metadata extracts and text extracts pulled from every crawl, and its own site states the archive has grown to over 300 billion pages since the project began in 2007. The mechanism is a monthly crawl cycle: CCBot requests a fresh set of URLs, Common Crawl packages the results, and the packaged crawl becomes downloadable by anyone with no login or fee. A site that wants its pages available for this kind of open reuse needs no configuration at all, since the default state is open.

How does blocking CCBot affect AI training corpora built from Common Crawl?

Blocking CCBot removes a site from the archive Common Crawl publishes, and that archive is a documented input to AI training work done outside Common Crawl itself. OpenAI's paper on GPT-3, Brown et al. 2020, lists "Common Crawl (filtered)" in its training data table at 410 billion tokens, 60 percent of the model's training mix, the largest single source in that table. The mechanism runs one direction only: a page CCBot never captures cannot appear in a Common Crawl snapshot, and a dataset another lab filters from that snapshot inherits the same absence, so a site decides its exposure to this specific route into AI training by deciding whether to let CCBot in at all.

Crawler	Operator	How it feeds AI training	robots.txt token
CCBot	Common Crawl	Publishes an open archive that outside labs filter into training sets	CCBot
GPTBot	OpenAI	Crawls directly to train OpenAI's own foundation models	GPTBot

How do you block CCBot in robots.txt?

Add a dedicated user agent block naming CCBot and disallow the paths you want excluded from every future crawl.

User-agent: CCBot
Disallow: /

That single block, published in this exact form on Common Crawl's own CCBot page, stops CCBot from requesting any page on the site going forward, though it does not remove pages a past crawl already captured. Pages already published in an earlier Common Crawl snapshot stay in that snapshot, so a new robots.txt rule only affects crawls from that point on. The same syntax accepts a narrower path, such as Disallow: /private/, when only part of a site needs to stay out of the archive.

How do you verify a request claiming to be CCBot?

Compare the request's source IP address against Common Crawl's published CCBot IP list at index.commoncrawl.org/ccbot.json before trusting the user agent string on its own, since Common Crawl's own documentation names spoofed CCBot traffic as a known problem. CCBot's IPv4 addresses carry working reverse DNS, giving a second way to confirm a request's origin independent of the header. A visitor sending the CCBot user agent string from an address outside that published range is not CCBot, whatever the header says.

How a robots.txt rule reaches an AI training set

CCBot crawls the open web for Common Crawl

Common Crawl publishes the result as a free, open archive

Outside labs, like OpenAI for GPT-3, filter that archive into training data

A User-agent: CCBot block removes a site before any of that happens

Should a site allow or block CCBot?

Allowing CCBot means adding no disallow rule for its user agent, which keeps a site inside a widely reused, no-cost research archive that Common Crawl has published monthly since 2007. Blocking it means adding the two-line rule above, which keeps a site out of that archive and out of any downstream dataset another lab filters from it, at the cost of losing whatever citation or discovery benefit comes from appearing in open research. The choice is the same one covered for Amazonbot and Applebot-Extended: a named crawler with a documented purpose, controlled by one line naming it in robots.txt.

Whether a page is even permitted to be crawled and reused this way at all is a separate legal question, covered in is web scraping legal, which applies to CCBot the same way it applies to any bot pulling content from a site it does not own. Once content does reach an AI answer through a route like this one, AI search brand monitoring tracks how a brand ends up represented in that answer, which is the outcome that matters once the crawl itself is long done.

Allow or block CCBot

Allow CCBot

• Stays inside the open archive Common Crawl has run since 2007
• No robots.txt change needed, since open access is the default
• Reusable by outside AI labs, universities, and competitors alike

Block CCBot

• One two-line robots.txt rule stops every future crawl
• Excluded from any dataset another lab filters from the archive, such as GPT-3's training mix
• Pages already captured in a past snapshot stay published regardless

Frequently Asked Questions

Does blocking CCBot remove my site from past Common Crawl archives?

No. A robots.txt rule only stops future crawls; it does not reach into a Common Crawl snapshot already published before the rule existed. Pages captured in an earlier crawl stay in that crawl's archive.

How do I know a request is really CCBot and not a spoofed crawler?

Match the request's source IP address against Common Crawl's published CCBot IP list at index.commoncrawl.org/ccbot.json. Common Crawl's own documentation states that other crawlers falsely identify themselves as CCBot, so the user agent header alone is not proof.

Does Common Crawl charge for access to its archive?

Common Crawl's own site describes the corpus as free and open, with no login or fee required to download a crawl. That open access is what lets outside labs build their own datasets from it.

Can I block CCBot from only part of my site?

Yes. A Disallow rule under the CCBot user agent block accepts a specific path, such as Disallow: /private/, so the rest of the site stays open to the same crawler.

Is Common Crawl the same thing as an AI company's own crawler?

No. Common Crawl publishes an open archive that any outside organization can download and filter, while a crawler like GPTBot crawls directly for its own operator, OpenAI, to train that operator's own models.

How frequently does Common Crawl publish a new crawl?

Common Crawl's own site states it adds 3 to 5 billion new pages each month as part of an archive that has run since 2007 and now holds over 300 billion pages. A robots.txt rule added at any point applies from that crawl cycle onward.

What formats does Common Crawl publish from each crawl?

Common Crawl publishes raw web page captures alongside metadata extracts and text extracts pulled from every monthly crawl. Each of these downloads for free, with no login required, once Common Crawl packages the results. The archive built from these releases already holds over 300 billion pages collected since 2007.

Does CCBot need any setup to start crawling a new site?

CCBot crawls a site by default, since the default state Common Crawl uses is open access with no configuration required. A site owner only needs to act to block CCBot, using the two-line robots.txt rule naming the CCBot user agent.

Why does OpenAI's GPT-3 paper matter when deciding whether to block CCBot?

OpenAI's paper on GPT-3, Brown et al. 2020, lists "Common Crawl (filtered)" as the largest single source in its training data table, at 410 billion tokens and 60 percent of the model's training mix. That filtered dataset traces back to pages CCBot captured, so a page CCBot never crawls cannot end up counted in a table like that one.

What is CCBot's reverse DNS check?

CCBot's published IPv4 addresses carry working reverse DNS entries, according to Common Crawl's own documentation. That gives a site a second way to confirm a request's origin, independent of matching the IP against Common Crawl's published list.