$ show togethercomputer/RedPajama-Data
This repository contains the data processing pipeline for the RedPajama-V2 dataset, a massive open dataset for training large language models. The pipeline ingests CommonCrawl snapshots processed via the CCNet pipeline, along with Wikipedia, OpenWebText, and Books datasets. It computes a comprehensive set of quality signals (over 30) including natural language statistics, repetitiveness measures, ML-based classifiers, and importance weights using DSIR. Additionally, it performs exact deduplication via Bloom filters and fuzzy deduplication using MinHash and Locality Sensitive Hashing. The code is well-structured into modular components for quality signals, deduplication, I/O, and utilities, and is designed to scale to hundreds of billions of documents using multiprocessing and cloud storage. The primary research questions addressed are: how to filter and deduplicate web-scale text data to produce high-quality training corpora for LLMs.
The LLM identified 4 files worth highlighting for reuse:
The Document class efficiently precomputes normalized content, word/line tokenization, n-grams (2,3,4), and hash features for DSIR importance weighting. It uses __slots__ for memory efficiency and lazy computation via property caching. The class is self-contained with no dependencies beyond nltk and internal utilities, making it extractable for any text processing pipeline that needs multi-level linguistic features.
Implements MinHash signature generation with configurable n-gram size and number of permutations. Uses numpy vectorization for efficient permutation hashing and generates banded signatures for multiple similarity thresholds (e.g., 0.7, 0.8, 0.9, 1.0). The optimal_param function computes band/row allocations. This component is standalone and can be reused in any near-deduplication project requiring LSH with Jaccard similarity.
Defines the abstract base class RPSBase for all quality signal functions. It enforces a naming convention (RPS_ prefix) and automatically derives the field name from the class name. Subclasses implement __call__ with a specific signal computation. This pattern allows dynamic registration of new signals via module introspection (registry_utils.py), making the signal system extensible and reusable across different data processing frameworks.
A simple, memory-efficient n-gram generator using a sliding window over an iterator. It yields tuples of consecutive items from any iterable sequence. The implementation avoids loading all data into memory and can be trivially reused for any n-gram or sequence chunking task with minimal modification.