$ show nlfiedler/fastcdc-rs
This is a pure Rust library implementing the FastCDC (Fast Content-Defined Chunking) algorithm, providing three independent implementations: ronomon (legacy, 31-bit), v2016 (original paper), and v2020 (optimized with 'rolling two bytes each time'). The v2020 module is the recommended entry point, offering in-memory chunking via `FastCDC` (Iterator), synchronous streaming via `StreamCDC`, and async streaming via `AsyncStreamCDC` (with tokio/futures feature flags). The core algorithm uses Gear hash tables derived from MD5 digests, normalized chunking with configurable levels, and an unrolled inner loop that processes six bytes per iteration for performance. The crate is deterministic: identical inputs always produce identical cut points and hashes, enforced by hardcoded fixture tests. It is designed for deduplication systems, backup software, and any application needing content-defined chunk boundaries. The codebase is well-structured, thoroughly tested, and includes a criterion benchmark suite with deterministic data generation for reproducible performance comparisons.
The LLM identified 2 files worth highlighting for reuse:
The core v2020 FastCDC implementation, including the `cut_gear_arr` function with a six-byte unrolled scan loop (`scan_region`) that processes three candidate pairs per iteration. This is the most performance-critical part of the crate, demonstrating non-obvious optimization by naming all six GEAR lookups upfront to allow LLVM to schedule independent loads. The function is self-contained (depends only on the GEAR arrays and mask parameters) and could be extracted as a standalone chunking primitive. Also includes `select_masks`, `get_gear_with_seed`, and the `FastCDC`/`StreamCDC` structs with clean iterator APIs.
A well-designed criterion benchmark suite that generates all test data deterministically from fixed seeds (SplitMix64 PRNG) across multiple content types (random, text, zeros, mixed). It benchmarks different chunker variants, code paths (iterator vs cut-loop vs stream), and average chunk sizes. The recommended workflow uses criterion's built-in baselines for before/after comparison. This file is a reusable pattern for any performance-sensitive Rust library needing reproducible microbenchmarks.