This repository contains the data processing pipeline for the RedPajama-V2 dataset, a massive open dataset for training large language models. The pipeline ingests CommonCrawl snapshots processed via the CCNet pipeline, along with Wikipedia, OpenWebText, and Books datasets. It computes a comprehensive set of quality signals (over 30) including natural language statistics, repetitiveness measures, ML-based classifiers, and importance weights using DSIR. Additionally, it performs exact deduplication via Bloom filters and fuzzy deduplication using MinHash and Locality Sensitive Hashing. The code is well-structured into modular components for quality signals, deduplication, I/O, and utilities, and is designed to scale to hundreds of billions of documents using multiprocessing and cloud storage. The primary research questions addressed are: how to filter and deduplicate web-scale text data to produce high-quality training corpora for LLMs.