What Is Common Crawl and How It Works

Common Crawl is a non-profit organization that provides a free and open dataset of web pages, allowing developers and researchers to access a vast amount of internet data.

This dataset is sourced from web crawlers that constantly scan the web for new content, resulting in a massive collection of web pages.

The data is then stored in a data repository, which is updated daily, allowing users to access the latest information.

Common Crawl's dataset is massive, containing over 25 terabytes of data, making it a valuable resource for those who need to analyze or extract insights from the web.

Common Crawl is a nonprofit organization that maintains an open repository of web crawl data, making it one of the most influential data sources on the web.

It's available to anyone, and its stated goal is to democratize access to web-scale data for research, AI, language modeling, SEO, and beyond. This means that anyone can access and use the data for their own purposes.

Suggestion: Data Commons

Common Crawl is responsible for crawling billions of pages each month, making a snapshot of the internet available to download in massive archives. This data is stored on Amazon S3, which can be queried to find specific sites.

The organization was founded in 2007 and first crawled the web in 2011, with its first crawl being completed in 2011. It was founded by Gil Elbaz, who is also the founder of Applied Semantics, the original AdSense.

The crawler used by Common Crawl is called CCBot/2.0, and it uses AWS-hosted IPs that are reverse DNSed to commoncrawl.org. This allows users to identify the crawler and its purpose.

Here are some key facts about Common Crawl:

Common Crawl's data is used by a wide range of organizations, including OpenAI, Meta, Hugging Face, and academic researchers, as well as SEO tools.

Common Crawl stores its data in Amazon S3 buckets under the commoncrawl public dataset, hosted in the AWS us-east-1 region (Northern Virginia).

The data is accessible through a specific URL, allowing users to access the vast amounts of web crawl data.

Common Crawl maintains a GitHub repository at github.com/commoncrawl, where users can find a variety of open-source projects to access and process the data.

Some notable repositories include:

These tools support a range of applications, from large-scale data analysis to NLP processing tasks.

Colossal Clean Corpus is a large dataset used for training language models, specifically the T5 language model series in 2019. It's a Google-developed version of Common Crawl, but with a cleaner and more curated approach.

The Colossal Clean Crawled Corpus, or C4 for short, has been used for various purposes, including web archiving and research initiatives.

Some concerns have been raised about copyrighted content in the C4, which is worth noting.

Recommended read: Web Programming Language Crossword

The data is stored in Amazon S3 buckets under the commoncrawl public dataset.

Common Crawl stores its data in the AWS us-east-1 region, specifically in Northern Virginia.

The data is accessible through this location.

Common Crawl has been collecting and archiving web data since 2008-2009, with the first crawl conducted from May 2008 through January 2009.

The size of the crawl data has increased significantly over the years, from 148 TiB in Winter 2013 to 454 TiB in December 2023.

Here's a breakdown of the crawl data size and number of pages crawled over the years:

The number of pages crawled has also increased, with the largest crawl in February/March 2024 containing over 3.16 billion pages.

Additional reading: See Website Archive

Common Crawl's data collection efforts have been ongoing since 2008, with the first crawl taking place from May 2008 to January 2009. This was the beginning of a long journey to collect and make available a vast amount of web data.

The organization has been hosting its archives on Amazon Web Services through the Public Data Sets program since 2012. This partnership has made it easier for users to access and utilize Common Crawl's data.

In 2012, Common Crawl started releasing metadata files and text output alongside .arc files. This change made it easier for users to work with the data. The organization received a significant donation of search engine metadata from blekko in December 2012, which helped improve the quality of the crawl.

Common Crawl began using the Apache Software Foundation's Nutch webcrawler in 2013, replacing its custom crawler. This change allowed for more efficient data collection. The organization also switched from .arc files to .warc files with its November 2013 crawl.

Here's a breakdown of the crawl data timeline:

The data shows a steady increase in the amount of data collected over the years, with some fluctuations.

You can use Amazon EMR to analyze petabytes of websites, as seen in Mark Litwintschik's work. This is just one example of how large-scale data analysis can be done.

Amazon Athena is another tool that can be used for this purpose, as demonstrated by Edward Ross's Common Crawl Index Athena project. It's a great way to process big data without having to manage infrastructure.

Sebastian Nagel's Index to WARC Files and URLs in Columnar Format is another example of how Amazon Athena can be used for data analysis. This project shows how to index WARC files and URLs in a columnar format.

Win Suen's Large-scale graph mining with Spark project also used Spark for data analysis. This project shows how to use Spark for large-scale graph mining.

Jader Dias's One click to download all the web pages you may want project used Amazon Athena and AWS Lambda to download web pages. This project is a great example of how to use these tools together.

Andres Riancho's Search the Common Crawl Using Lambda Functions project is another example of using AWS Lambda for data processing. This project shows how to use Lambda functions to search the Common Crawl dataset.

Here are some examples of how to use Common Crawl data:

Common Crawl's Crawler Bot is quite unique. It crawls about once a month, focusing on providing a foundational dataset rather than full indexing or search services.

This approach allows researchers, developers, and companies to build on top of it. Common Crawl uses a Nutch based crawler, which is a testament to its commitment to open-source technology.

Apache Nutch is an open-source web crawler built on top of Apache Hadoop.

For another approach, see: Apache Axis2

Bots are fascinating tools that help us scrape and collect data from the web.

Crawler Bot is a type of bot that crawls the web about once a month to collect data. It's not designed to provide full indexing or search services, but rather serves as a foundational dataset that others can build on.

Common Crawl is an example of a Crawler Bot, and it's used by researchers, developers, and companies to gather web data.

Nutch is an open-source web crawler built on top of Apache Hadoop. It's used by CommonCrawl, which is a great example of its capabilities.

Apache Hadoop is a powerful tool for big data processing, and Nutch leverages its strengths to crawl the web efficiently.

Common Crawl identifies itself with the following user-agent:

It's worth noting that you might also see this user-agent referred to as a "bot" or "crawler".

Common Crawl uses a Nutch based crawler, built on top of Apache Hadoop, which is an open-source web crawler.

They don't publish a static list of IPs, but all their crawlers resolve to identifiable AWS IPs, often in the ccbot.commoncrawl.org domain.

Site owners can verify requests by doing a reverse DNS lookup on the IP to confirm it ends with commoncrawl.org.

Common Crawl also offers indexed search and filtering tools.

Common Crawl is recognized for its contributions to the field of web data science. The Norvig Web Data Science Award is a testament to this recognition.

The award is a competition open to students and researchers in Benelux. It's sponsored by Common Crawl in collaboration with SURFsara.

The award is named after Peter Norvig, a notable figure who also chairs the judging committee for the award.

Intriguing read: Health Web Science

If your site is publicly accessible and not specifically blocking Common Crawl, you're contributing to the world's open web memory bank – whether you meant to or not.

Curious to learn more? Check out: What Browesr Does Not save as Webp

If you care about who's copying or analyzing your site at scale, you should care about Common Crawl. Your content may be part of the training data for large language models used by companies like OpenAI, Google, and Meta.

Common Crawl's data is regularly used for link graph analysis, competitive intelligence, and scraping tools, which can have SEO implications for your site. A post from two hours ago on this site is already in CommonCrawl, demonstrating how quickly your content can be updated in the dataset.

Here are some key facts to keep in mind:

Backlinks and Link Structure are crucial for a website's online presence. Common Crawl publishes web graphs that represent hyperlink relationships between websites.

These graphs are available at both the host and domain levels, showing how different sites link to one another. The most recent release, covering May-July 2025, includes a staggering 481.6 million nodes and 3.4 billion edges in its host-level graph.

A different take: Link after Video Playback Html

The data used to construct these graphs comes from WAT files, which contain metadata about crawled pages, including outlinks. The WAT files are processed using tools like cc-pyspark and the WebGraph framework to generate the final graph structures.

Common Crawl's web graphs are used by sites like SemRush, Majestic, and Inlinks for further analysis and usage in their services. The domain-level graph in the recent release has 209.5 million nodes and 2.6 billion edges.

Related reading: Which Web Browser Is Most Used Worldwide

If your site is publicly accessible and not specifically blocking Common Crawl, you're contributing to the world's open web memory bank – whether you meant to or not. For developers and researchers, it's a goldmine.

Your content can be crawled and stored in Common Crawl's archives in a matter of minutes. In fact, a post from two hours ago on this site is already in CommonCrawl.

Common Crawl data is regularly used for link graph analysis, competitive intelligence, and scraping tools, which can have SEO implications for your site. This means your content is being analyzed and potentially used by others without your explicit permission.

Discover more: Content Protection Network

Once crawled, your content can remain in the dataset for years – even if removed from your site. This is a long time for your content to be out there, and it's worth considering the implications of this data permanence.

Here's a brief rundown of the potential uses of Common Crawl data:

Common Crawl offers a range of tools for accessing and processing their web crawl data. These tools are open source and can be found on their GitHub repository.

The organization maintains a variety of projects, including cc-pyspark, a Python and Spark-based toolkit for processing CC data.

Some notable repositories include:

These tools support a range of applications, from large-scale data analysis to NLP processing tasks.