$ show dragonflydb/dragonfly
DragonflyDB is a high-performance, Redis-compatible in-memory data store written in C++. It implements the Redis wire protocol (RESP) and Memcached protocol, providing drop-in replacement capabilities with significantly higher throughput and lower latency, especially on multi-core machines. The architecture is based on a shared-nothing, multi-threaded design using fibers (boost.fibers) and a custom proactor event loop (epoll/io_uring). Data is partitioned across shards, each with its own thread, and transactions use a VLL-based locking scheme to minimize contention. The system supports replication via a write-ahead journal, cluster mode with slot migration, persistence via RDB snapshots, and tiered storage for offloading cold data to SSD. It also includes a full-text search engine with vector similarity (HNSW), JSON document store, Lua scripting, ACL-based security, and probabilistic data structures (Bloom, Cuckoo, Count-Min Sketch, Top-K). The codebase demonstrates advanced engineering in memory management (custom allocators, SIMD probing, compact object encoding) and concurrency (fiber-aware synchronization, RCU patterns). While primarily an application, it contains research-grade data structures and algorithms, giving it a hybrid character between production service and systems research project.
The LLM identified 4 files worth highlighting for reuse:
Provides a portable `movemask` operation across SIMD instruction sets (AVX2, SSE, NEON) and a scalar fallback. This is used for probing open-addressing hash tables (like OAH) to quickly find matching slots. The abstraction is clean, header-only, and can be reused in any SIMD-accelerated hash table or search implementation.
Implements a lightweight intent lock with SHARED and EXCLUSIVE modes, used for transaction ordering in Dragonfly's scheduling. The lock supports upgrading and downgrading, and is designed for fiber-based concurrency. It is self-contained and could be extracted for use in other concurrent systems requiring fine-grained lock modes.
A fluent argument parser for Redis command arguments, supporting typed extraction (int, double, string) with validation rules (e.g., positive, range, non-empty). It provides a clean API for command handlers to parse and validate arguments in a declarative style. Reusable in any Redis-compatible server or library that needs to parse RESP arrays.
Implements the HeavyKeeper Top-K algorithm, combining a Count-Min Sketch for frequency estimation with a min-heap for tracking the top K elements. Supports insert, query, and merge operations, and uses PMR allocators for memory tracking. This is a self-contained data structure that can be reused in any application needing approximate top-K tracking (e.g., trending topics, hot keys).