Repositories list

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •9292 forks•22 stars•00 issues•22 pull requests•Updated Oct 2, 2026Oct 2, 2026

• cc-webgraph

  Public
  Tools to construct and process Common Crawl webgraphs

  pagerankwebgraphcommoncrawl+ 3

  pagerankwebgraphcommoncrawlcommon-crawlcentrality-measureswebgraph-framework

  Java

  •

  Apache License 2.0

  •77 forks•115115 stars•77 issues•77 pull requests•Updated Oct 1, 2026Oct 1, 2026

• cc-warc-examples

  Public
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

  javahadoopmapreduce+ 1

  javahadoopmapreducecommoncrawl

  Java

  •

  MIT License

  •4444 forks•3838 stars•00 issues•00 pull requests•Updated Oct 1, 2026Oct 1, 2026

• ia-hadoop-tools

  Public
  Web archiving tools on Hadoop

  Java

  •2727 forks•44 stars•22 issues•11 pull request•Updated Oct 1, 2026Oct 1, 2026

• ia-web-commons

  Public
  Web archiving utility library

  cdx-fileswarc-fileswat-files

  cdx-fileswarc-fileswat-files

  Java

  •

  Apache License 2.0

  •7878 forks•1212 stars•44 issues•11 pull request•Updated Oct 1, 2026Oct 1, 2026

• cc-index-table

  Public
  Index Common Crawl archives in tabular format

  sqlsparkcolumnar-storage+ 3

  sqlsparkcolumnar-storageaws-athenaapache-parquetcommoncrawl

  Java

  •

  Apache License 2.0

  •1717 forks•132132 stars•66 issues•88 pull requests•Updated Oct 1, 2026Oct 1, 2026

• news-crawl

  Public
  News crawling with StormCrawler - stores content as WARC

  crawlernewsweb-crawler+ 5

  crawlernewsweb-crawlerapache-stormwarccommoncrawlcommon-crawlstorm-crawler

  Java

  •

  Apache License 2.0

  •4242 forks•379379 stars•1313 issues•55 pull requests•Updated Oct 3, 2026Oct 3, 2026

• nutch

  Public
  Common Crawl fork of Apache Nutch

  javabig-datahadoop+ 2

  javabig-datahadoopweb-crawlercommoncrawl

  Java

  •

  Apache License 2.0

  •1.3k1.3k forks•4343 stars•88 issues•00 pull requests•Updated Aug 13, 2026Aug 13, 2026

• language-detection-cld2

  Public
  Natural language detection, Java bindings for CLD2

  natural-languagelanguage-detectionlanguage-identification

  natural-languagelanguage-detectionlanguage-identification

  Java

  •

  Apache License 2.0

  •22 forks•2020 stars•33 issues•11 pull request•Updated Aug 2, 2026Aug 2, 2026

• whirlwind-java

  Public
  A whirlwind tour of Common Crawl's data using Java

  Java

  •

  Apache License 2.0

  •00 forks•33 stars•11 issue•00 pull requests•Updated Apr 20, 2026Apr 20, 2026

• commoncrawl-crawler

  Public archive
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)

  archivedinactive

  archivedinactive

  Java

  •

  GNU General Public License v3.0

  •6464 forks•226226 stars•00 issues•00 pull requests•Updated Dec 22, 2022Dec 22, 2022

• Teneo

  Public
  Sebastian Spiegler's statistics of the Common Crawl corpus 2012

  archivedinactive

  archivedinactive

  Java

  •99 forks•00 stars•00 issues•00 pull requests•Updated Oct 2, 2017Oct 2, 2017

• example-warc-java

  Public
  archivedinactive

  archivedinactive

  Java

  •1111 forks•5050 stars•00 issues•00 pull requests•Updated Feb 22, 2017Feb 22, 2017

• commoncrawl-examples

  Public archive
  A library of examples showing how to use the Common Crawl corpus (2008-2012, ARC format)

  archivedinactive

  archivedinactive

  Java

  •4343 forks•6767 stars•00 issues•22 pull requests•Updated Aug 5, 2016Aug 5, 2016

• example-apprankings

  Public
  Java

  •22 forks•22 stars•00 issues•00 pull requests•Updated Jan 15, 2013Jan 15, 2013

• example-javascriptusage

  Public
  Java

  •11 fork•00 stars•00 issues•00 pull requests•Updated Jan 15, 2013Jan 15, 2013

• example-companyfootprints

  Public
  Java

  •22 forks•00 stars•00 issues•00 pull requests•Updated Jan 15, 2013Jan 15, 2013

• example-europeanjob

  Public
  Java

  •11 fork•11 star•00 issues•00 pull requests•Updated Jan 15, 2013Jan 15, 2013

• example-languageentropy

  Public
  Java

  •11 fork•00 stars•00 issues•00 pull requests•Updated Jan 15, 2013Jan 15, 2013

• example-babel2012

  Public
  Java

  •44 forks•00 stars•00 issues•00 pull requests•Updated Jan 13, 2013Jan 13, 2013

• example-wikientities

  Public
  Linking Entities in CommonCrawl Dataset onto Wikipedia Concepts

  Java

  •1313 forks•44 stars•00 issues•00 pull requests•Updated Sep 5, 2012Sep 5, 2012