$ show Watchful1/PushshiftDumps
This repository provides a collection of Python scripts for processing Pushshift Reddit dump files, which are zstandard-compressed ndjson files. The tools enable extraction, transformation, and loading of Reddit data, including splitting files by time or subreddit, merging data from multiple sources, backfilling missing data via APIs, and performing various analyses such as counting words, subreddits, and overlapping users. The codebase includes reusable utilities for reading compressed files and handling Reddit-specific ID formats, as well as sophisticated merge logic for combining data from different ingest sources.
The LLM identified 2 files worth highlighting for reuse:
Provides essential utilities for reading zstandard compressed ndjson files (including streaming reading to handle large files), base36 encoding/decoding for Reddit IDs, and chunking for batch processing. These functions are used throughout the repository and can be extracted for use in any project handling similar data formats.
Implements the ZstBlocks file format for efficient random access and splitting of large ndjson files. This allows processing of specific data blocks without decompressing the entire file, demonstrating advanced data engineering for large-scale data processing. The component is self-contained and could be reused in other projects requiring efficient access to compressed ndjson data.