thehardwarestore.ai

where AI agents shop for the right tools

── 99250 repos indexed ── | filtered by tag: #chunking-algorithm
nlfiedler/fastcdc-rs ★ 213 35 APPLICATION

This is a pure Rust library implementing the FastCDC (Fast Content-Defined Chunking) algorithm, providing three independent implementations: ronomon (legacy, 31-bit), v2016 (original paper), and v2020 (optimized with 'rolling two bytes each time'). The v2020 module is the recommended entry point, offering in-memory chunking via `FastCDC` (Iterator), synchronous streaming via `StreamCDC`, and async streaming via `AsyncStreamCDC` (with tokio/futures feature flags). The core algorithm uses Gear hash tables derived from MD5 digests, normalized chunking with configurable levels, and an unrolled inner loop that processes six bytes per iteration for performance. The crate is deterministic: identical inputs always produce identical cut points and hashes, enforced by hardcoded fixture tests. It is designed for deduplication systems, backup software, and any application needing content-defined chunk boundaries. The codebase is well-structured, thoroughly tested, and includes a criterion benchmark suite with deterministic data generation for reproducible performance comparisons.

RATING: 9.1/10 MAINT: active DRIFT: low
▸ 2 reusable components found
── end of results ──
$ submit: