RocksDB is a high-performance, embeddable, persistent key-value store library written in C++. It uses an LSM-tree architecture with memtables, sorted string tables (SSTs), multi-level compaction, write-ahead logging (WAL), and advanced caching (LRU, Clock, tiered secondary caches). The codebase is exceptionally well-engineered, with extensive use of templates, platform abstractions, and custom allocators to achieve high throughput and low latency on flash storage. It is not a standalone application but a library consumed by databases (e.g., MyRocks, Cassandra), storage engines, and other infrastructure. The repository includes a comprehensive public API under include/rocksdb/, a C API, Java bindings via JNI, command-line tools (ldb, sst_dump, db_bench), and a fuzz testing suite. The project is actively maintained by Meta and the open-source community, with frequent commits, rigorous CI, and performance benchmarks.
Valkey is a high-performance in-memory data structure server, forked from Redis under a BSD-3-Clause license. It serves as a key-value store with support for strings, hashes, lists, sets, sorted sets, streams, geospatial indexes, and more. The architecture is event-loop-driven (single-threaded main loop with background I/O threads for persistence, replication, and lazy freeing). It supports clustering (up to 16384 slots), replication (leader-follower with partial resync), persistence (RDB snapshots, AOF logs), Lua scripting (EVAL/FUNCTION), a pluggable module API, TLS, RDMA, and advanced features like client-side caching, hot-key detection, and replication throttling. Key entry points include the `valkey-server` daemon, `valkey-cli` client, `valkey-benchmark` load generator, and `valkey-check-rdb`/`valkey-check-aof` integrity tools. External dependencies include jemalloc (default allocator), Lua 5.1, LZ4, Zstd (optional), OpenSSL (TLS), and RDMA libraries (optional). The codebase is actively maintained with extensive unit and integration tests, and demonstrates high craftsmanship in its lock-free queues, streaming compression, and data structure implementations.
bbolt is a production-grade, embedded key-value store for Go, forked from Bolt and inspired by LMDB. It provides ACID transactions, memory-mapped zero-copy reads, and a B+tree data model with buckets and cursors. The core library is used as the storage engine for etcd, InfluxDB, Consul, and other high-throughput systems. The repository also includes a full-featured CLI tool (`bbolt`) for database inspection, integrity checking, benchmarking, and surgical repair (e.g., page copying, freelist reconstruction). The architecture is a pure-Go library with minimal dependencies, relying on platform-specific syscalls for mmap, flock, and mlock. The codebase demonstrates strong engineering practices: clean abstractions (swappable freelist implementations via an interface), extensive test coverage including fault injection and robustness tests against filesystem failures, and a well-organized CLI command structure using Cobra. While primarily a library, the CLI makes it also a standalone application for database administration.
redb is a high-performance, ACID-compliant embedded key-value store library written in Rust. It provides a B-tree-based storage engine with copy-on-write semantics, MVCC concurrency control, and support for typed tables, multimaps, savepoints (ephemeral and persistent), compaction, and crash recovery. The library exposes a clean public API centered around `Database`, `TableDefinition`, `WriteTransaction`, `ReadTransaction`, and `ReadableTable` traits, with optional derive macros for user-defined key/value types. It is designed for single-writer, multiple-reader access and supports multi-process concurrency via byte-range file locking (experimental). The codebase is extensively tested with integration tests, fuzzing, and benchmarks comparing against LMDB, RocksDB, SQLite, and fjall. Architecturally, redb uses a copy-on-write B+tree for all tables (data, system, freed pages) with a buddy allocator for page management and a region-based file layout. The page store includes a striped LRU read cache, write buffer, and dynamic cache partitioning. Commit strategies range from non-durable (no fsync) to 1PC+C (default) and 2PC (secure). Crash recovery relies on XXH3-128 checksums and optional quick-repair via persistent allocator state. The library is `no_std`-compatible with feature-gated std-dependent parts, and includes a custom spin-lock synchronization module for embedded targets. Key entry points include the `Database::builder()` and `Database::open()` methods, `TableDefinition::new()` for schema definition, and transaction methods like `WriteTransaction::commit()` and `ReadTransaction::open_table()`. The `StorageBackend` trait allows pluggable storage (file, in-memory, custom) and is used extensively in tests for fault injection and crash simulation. External dependencies are minimal (standard library, optional chrono/uuid), with benchmark crates depending on heed, rocksdb, fjall, and rusqlite for comparison. The repository demonstrates strong engineering craftsmanship with clean abstractions, careful handling of edge cases (max keys, large values, corruption), and a well-documented design document (`docs/design.md`). The codebase is actively maintained with a stable file format and backward compatibility guarantees.
BuntDB is a low-level, in-memory, key/value store library in Go. It supports custom indexing (B-tree and R-tree), geospatial queries, JSON indexing, TTL expiration, and ACID transactions with multiple readers / single writer. Persistence is via an append-only file (AOF) with automatic background shrinking. The library is designed for high performance and embeddability, and is used by projects requiring a fast, reliable embedded database. The codebase is a single package with a well-structured API, comprehensive tests, and minimal dependencies.
This repository is a Redis-compatible in-memory data structure server with significant custom extensions, including vector similarity search via HNSW (vector-sets module), rate limiting (GCRA), key eviction protection (BLESS), and a comprehensive module system. The architecture is a single-threaded event loop (ae) with optional IO threads, supporting a wide range of data structures: strings, hashes, lists, sets, sorted sets, streams, geospatial indexes, HyperLogLog, and bitfields. Persistence is provided via RDB snapshots and AOF logs, with replication and cluster support. The codebase is a fork/derivative of Redis/Valkey, incorporating additional features not present in upstream, such as hash field expiration (ebuckets/estore), per-key metadata (keymeta), and a Fenwick tree-based key selection (kvstore). Key entry points include the redis-server daemon, redis-cli, redis-benchmark, and module loading via the redismodule.h API. External dependencies include jemalloc, hiredis, Lua 5.1, linenoise, tre, xxhash, OpenSSL, and optional ZSTD compression. The system is designed for high throughput and low latency, with extensive use of custom memory management (zmalloc) and data structure optimizations.
RoseDB is a lightweight, fast, and reliable key-value storage engine implemented in Go, based on the Bitcask storage model. It provides a simple embedded database library with operations like Put, Get, Delete, batch writes, TTL/expiry, key watching, and merge (compaction). The architecture uses a Write-Ahead Log (WAL) for persistence and an in-memory BTree index for fast lookups. The codebase is well-structured, with clean separation of concerns: the DB layer, index abstraction, record encoding, and utility functions. It demonstrates strong engineering practices including thread safety via sync.RWMutex, object pooling for performance, and a pluggable index interface. The repository includes comprehensive tests, benchmarks, and examples. It is actively maintained with recent commits and a responsive community.
SortMeRNA is a bioinformatics application designed to filter ribosomal RNA (rRNA) sequences from metatranscriptomic or total RNA sequencing data. It employs a seed-and-extend alignment strategy using a burst trie index for efficient k-mer lookup and Smith-Waterman for alignment extension, coupled with statistical significance testing (E-value) to distinguish rRNA from non-rRNA reads. The tool processes input in parallel via a thread pool and supports both single-end and paired-end reads in FASTA/FASTQ format (including gzipped), producing aligned and non-aligned output files in multiple formats (FASTA, FASTQ, SAM, BLAST-like) along with optional OTU clustering and alignment reports. The application follows a modular architecture with distinct components for input handling (Readfeed), reference indexing (Index/IndexDB), parallel processing (Processor/ThreadPool), alignment computation, and output reporting (various Report* modules). Key features include a checkpoint/resume mechanism implemented using a RocksDB-based key-value store (KVDB) to enable recovery from interruptions, and a portable SIMD layer (SSE2NEON) that allows the same code to leverage vectorized instructions on both x86 and ARM architectures. The build system is managed by CMake with conda-based dependency handling, ensuring reproducibility across platforms. Despite its complexity due to the integration of multiple third-party libraries (zlib, RocksDB, BBHash, Parasail, ALP) and the need for careful version management, the project demonstrates active maintenance with recent commits and a structured release history. The codebase exhibits several well-crafted, reusable components such as the thread pool, KVDB wrapper, and SSE2NEON compatibility layer, which could be extracted for use in other projects with minimal modification. The tool is primarily used in microbiome and transcriptomics workflows for rRNA depletion, a critical preprocessing step in metatranscriptomic analysis.
openacid/slim is a Go library for building memory-efficient, serializable trie indexes and sparse arrays, primarily designed for in-memory indexing of large on-disk key-value data. The core artifact is SlimTrie, a compressed trie that achieves roughly 11 bits per key while supporting fast Get, range queries, prefix scans, and versioned marshal/unmarshal. The repository also contains a standalone sparse-array package and a type-safe encoder framework, with extensive benchmarks comparing against Go maps and btree indexes. Its architecture is modular: the trie package defines the index structure, the array package provides bitmap-backed dense/sparse storage primitives, and the encode package abstracts value serialization with generated concrete implementations.
LevelDB is a high-performance, embedded key-value storage library written in C++. It implements a Log-Structured Merge-Tree (LSM) architecture, organizing data into sorted tables across multiple levels to optimize write throughput and read latency. The codebase provides a stable C++ API with a C wrapper, supporting atomic batch operations, transient snapshots, and forward/backward iteration. It is designed for single-process access with no builtin client-server support, making it ideal for embedded use cases where low-latency access and crash recovery are critical. The architecture is monolithic, consisting of core database logic in the `db/` directory, table management in `table/`, and utility functions in `util/`. Platform abstraction is handled via the `port/` directory, isolating OS-specific file and thread operations. The library relies on external dependencies for compression (Snappy, Zstd) and testing (GTest), managed via CMake and submodules. It emphasizes durability through Write-Ahead Logging (WAL) and robust recovery mechanisms that validate manifests and replay logs on startup. While the codebase demonstrates exceptional craftsmanship in concurrency control and data structures, it is receiving very limited maintenance, accepting only critical bug fixes. The repository requires C++17 and CMake 3.22, which may limit compatibility with older toolchains. The C API requires careful ABI management, and platform-specific implementations in `util/env_posix.cc` and `util/env_windows.cc` risk divergence over time.
mini-redis is an incomplete, idiomatic implementation of a Redis client and server built with Tokio, serving as a teaching resource for asynchronous Rust patterns. The server supports key-value operations (GET, SET) with optional TTL, pub/sub (PUBLISH, SUBSCRIBE, UNSUBSCRIBE), and PING. It demonstrates graceful shutdown, connection limiting via a semaphore, and key expiration using a background task. The client library provides async (Client), blocking (BlockingClient), and buffered (BufferedClient) clients, with a Subscriber type for pub/sub. The codebase is well-structured, extensively commented, and showcases patterns such as frame-level protocol parsing, shared state with std::sync::Mutex, message-passing for multiplexing, and broadcast channels for shutdown signals. It is not intended for production use but is a high-quality reference for learning Tokio and async Rust.
This is the official NATS C client library (nats.c), a production-grade C library providing full support for the NATS messaging system. It implements core publish/subscribe, request-reply, queue groups, and advanced features including JetStream (streams, consumers, key-value store, object store), a microservices framework, and legacy NATS Streaming support. The library is designed for high performance and reliability, with built-in reconnection, TLS/SSL, authentication (NKeys, JWT), and event loop integration (libevent, libuv). It exposes a comprehensive C API with hundreds of functions, and includes extensive examples and documentation. The codebase is well-structured with platform abstractions for Unix and Windows, and internal modules for protocol parsing, connection management, threading, and memory management. While the library is actively maintained and widely used, it carries some technical debt from legacy Streaming support and a large, tightly-coupled internal architecture.
Kronotop is a distributed database system built on FoundationDB that provides Redis-compatible (RESP2/RESP3) interfaces for multiple data models including key-value (stash), document (bucket with BSON storage and querying), and sorted sets (zmap). It features ACID transactions, secondary indexing (including vector search via HNSW), clustering, sharding, replication, and namespace isolation for multi-tenancy. The architecture is modular with clear separation between network handling (Netty), protocol processing, service layers (transaction, session, data services), and storage (volume-based segment store). Key entry points include the server entry point (KronotopInstanceStarter), CLI (kronotop-cli), and control tool (kronotop-ctl). The system relies on external dependencies such as FoundationDB, Netty, Jackson, Lettuce, Guava, JVector, and Typesafe Config. The codebase demonstrates strong engineering practices with reusable components like versionstamp utilities, retry mechanisms, and protocol handling.
Sled is an embedded key-value store written in Rust that provides ACID transactions, crash safety, and high concurrency. It uses a lock-free architecture based on a lock-free pagecache, lock-free log, and lock-free B+ tree with epoch-based memory reclamation. The database supports multiple keyspaces, watch prefixes, merge operators, and a crash-safe monotonic ID generator. Currently in active development with a major rewrite underway (komora project), it is alpha software with known instability and frequent breaking changes, but demonstrates advanced concurrent data structure techniques.
Sophia is an embedded transactional key-value/row storage engine written in C, designed for high performance with ACID guarantees, MVCC (Multiversion Concurrency Control), and serializable snapshot isolation. Its architecture is inspired by LSM-trees but uses a unique node/branch/region model with O(1) worst-case set/delete and amortized O(branches) get. The system supports multiple databases, secondary indexes, custom comparators, upsert callbacks, compression (LZ4, Zstd), hot backup, online compaction/checkpointing, write-ahead logging (WAL), and crash recovery with fault injection. The codebase is modular with layers: a portable standard library (allocators, VFS, data structures, iterators, compression filters), a format/schema layer, an index layer, a transaction manager with deadlock detection, a scheduler for background jobs, and a public C API (sp_*). It includes an extensive test suite with unit, functional, crash, multithread, and hermitage isolation tests. The project is no longer actively developed, and the bundled third-party compression libraries are outdated, but the design and abstractions remain of educational and archival value.