RocksDB is a high-performance, embeddable, persistent key-value store library written in C++. It uses an LSM-tree architecture with memtables, sorted string tables (SSTs), multi-level compaction, write-ahead logging (WAL), and advanced caching (LRU, Clock, tiered secondary caches). The codebase is exceptionally well-engineered, with extensive use of templates, platform abstractions, and custom allocators to achieve high throughput and low latency on flash storage. It is not a standalone application but a library consumed by databases (e.g., MyRocks, Cassandra), storage engines, and other infrastructure. The repository includes a comprehensive public API under include/rocksdb/, a C API, Java bindings via JNI, command-line tools (ldb, sst_dump, db_bench), and a fuzz testing suite. The project is actively maintained by Meta and the open-source community, with frequent commits, rigorous CI, and performance benchmarks.
MinIO is a high-performance, S3-compatible object storage server implemented in Go. It is a monolithic application that provides a full-featured object storage system supporting erasure coding, bitrot protection, IAM, replication, lifecycle management, and a web console. The codebase is organized around a central `cmd` package containing the server implementation, with extensive `internal` packages for subsystems like erasure coding, encryption, event notifications, and distributed locking. The architecture follows a distributed systems pattern where nodes communicate via custom REST protocols and a proprietary grid messaging system for internode communication. The system implements a sophisticated storage engine using Reed-Solomon erasure coding with support for multiple storage classes, bitrot detection via HighwayHash, and strong consistency guarantees. It supports a wide range of protocols including S3 API, FTP/SFTP, and S3 Select for SQL queries on stored objects. The codebase demonstrates advanced Go patterns including extensive use of build tags for platform-specific code, code generation for serialization (msgp), and a plugin-based architecture for external identity providers and KMS integration. Key architectural components include the erasure coding subsystem (`cmd/erasure*.go`), the object API layer (`cmd/object-api*.go`), the IAM system (`cmd/iam*.go`, `cmd/auth-handler.go`), and the distributed systems infrastructure (`internal/grid`, `internal/dsync`). The codebase shows significant investment in operational features including comprehensive metrics exposure via Prometheus, audit logging, batch jobs for large-scale operations, and extensive debugging tools. The project maintains high code quality with extensive testing, CI/CD pipelines, and support for multiple deployment modes including standalone, distributed, Kubernetes, and various cloud platforms.
redb is a high-performance, ACID-compliant embedded key-value store library written in Rust. It provides a B-tree-based storage engine with copy-on-write semantics, MVCC concurrency control, and support for typed tables, multimaps, savepoints (ephemeral and persistent), compaction, and crash recovery. The library exposes a clean public API centered around `Database`, `TableDefinition`, `WriteTransaction`, `ReadTransaction`, and `ReadableTable` traits, with optional derive macros for user-defined key/value types. It is designed for single-writer, multiple-reader access and supports multi-process concurrency via byte-range file locking (experimental). The codebase is extensively tested with integration tests, fuzzing, and benchmarks comparing against LMDB, RocksDB, SQLite, and fjall. Architecturally, redb uses a copy-on-write B+tree for all tables (data, system, freed pages) with a buddy allocator for page management and a region-based file layout. The page store includes a striped LRU read cache, write buffer, and dynamic cache partitioning. Commit strategies range from non-durable (no fsync) to 1PC+C (default) and 2PC (secure). Crash recovery relies on XXH3-128 checksums and optional quick-repair via persistent allocator state. The library is `no_std`-compatible with feature-gated std-dependent parts, and includes a custom spin-lock synchronization module for embedded targets. Key entry points include the `Database::builder()` and `Database::open()` methods, `TableDefinition::new()` for schema definition, and transaction methods like `WriteTransaction::commit()` and `ReadTransaction::open_table()`. The `StorageBackend` trait allows pluggable storage (file, in-memory, custom) and is used extensively in tests for fault injection and crash simulation. External dependencies are minimal (standard library, optional chrono/uuid), with benchmark crates depending on heed, rocksdb, fjall, and rusqlite for comparison. The repository demonstrates strong engineering craftsmanship with clean abstractions, careful handling of edge cases (max keys, large values, corruption), and a well-documented design document (`docs/design.md`). The codebase is actively maintained with a stable file format and backward compatibility guarantees.
RustFS is a production-grade, S3-compatible distributed object storage server implemented in Rust. The core system provides erasure-coded storage with healing, bitrot detection, bucket versioning, object locking, lifecycle management, replication (bucket and site), tiering to remote S3-compatible backends, KMS integration, IAM/OIDC/Keystone authentication, audit logging, event notifications, and an extensive observability layer. It also exposes non-S3 protocols including OpenStack Swift, SFTP, FTPS, and WebDAV, plus an admin/console API and an embedded server library for in-process testing and integration. The repository is a large Cargo workspace of roughly 50 crates, with the main `rustfs` binary orchestrating startup, HTTP/gRPC servers, background services, and protocol servers, while domain subsystems such as `ecstore`, `heal`, `scanner`, `replication`, `notify`, `kms`, `iam`, `obs`, and `targets` are isolated behind trait-based contracts and runtime-boundary modules.
Apache Cassandra is a distributed, wide-column NoSQL database designed for high availability, horizontal scalability, and fault tolerance. The codebase implements a peer-to-peer cluster architecture with consistent hashing for data placement and tunable replication. The system provides the Cassandra Query Language (CQL) as its primary interface, with a native binary protocol for client communication. Key subsystems include the storage engine (SSTables, memtables, commit log), distributed consensus protocols (legacy Paxos and the newer Accord transactional protocol), cluster metadata management (TCM replacing gossip), and secondary indexing (SASI and Storage-Attached Index SAI). The codebase also includes operational tooling (nodetool CLI, cqlsh shell, stress testing tool, sstable utilities) and extensive testing infrastructure (unit, distributed in-JVM, deterministic simulator, property-based, and microbenchmarks). The architecture follows a modular design with clear separation of concerns: transport layer (Netty-based NIO), query processing (CQL parser, statement execution, result set building), storage (commit log for durability, memtables for writes, SSTables for reads, compaction strategies), and distributed coordination (gossip for failure detection, streaming for bootstrap/repair, hinted handoff for availability). The newer Accord protocol introduces a transactional layer on top of the existing storage engine, enabling ACID transactions across partitions. The codebase is predominantly Java with a Python-based CQL shell (cqlsh) and extensive shell-based build and CI automation. Key entry points include the Cassandra server daemon (started via bin/cassandra), the nodetool CLI (bin/nodetool) for administration, cqlsh (bin/cqlsh) for interactive CQL, and various sstable utilities (sstableloader, sstablescrub, etc.). External dependencies include Netty for networking, Dropwizard Metrics for instrumentation, Picocli for CLI parsing, and various Java libraries for compression, encryption, and serialization. The build system uses Ant with Maven dependency resolution, and the project includes Dockerfiles for containerized builds and testing. The repository also contains formal verification artifacts (TLA+ models) and a deterministic simulation framework for testing distributed correctness, reflecting a strong emphasis on reliability.
RoseDB is a lightweight, fast, and reliable key-value storage engine implemented in Go, based on the Bitcask storage model. It provides a simple embedded database library with operations like Put, Get, Delete, batch writes, TTL/expiry, key watching, and merge (compaction). The architecture uses a Write-Ahead Log (WAL) for persistence and an in-memory BTree index for fast lookups. The codebase is well-structured, with clean separation of concerns: the DB layer, index abstraction, record encoding, and utility functions. It demonstrates strong engineering practices including thread safety via sync.RWMutex, object pooling for performance, and a pluggable index interface. The repository includes comprehensive tests, benchmarks, and examples. It is actively maintained with recent commits and a responsive community.
LevelDB is a high-performance, embedded key-value storage library written in C++. It implements a Log-Structured Merge-Tree (LSM) architecture, organizing data into sorted tables across multiple levels to optimize write throughput and read latency. The codebase provides a stable C++ API with a C wrapper, supporting atomic batch operations, transient snapshots, and forward/backward iteration. It is designed for single-process access with no builtin client-server support, making it ideal for embedded use cases where low-latency access and crash recovery are critical. The architecture is monolithic, consisting of core database logic in the `db/` directory, table management in `table/`, and utility functions in `util/`. Platform abstraction is handled via the `port/` directory, isolating OS-specific file and thread operations. The library relies on external dependencies for compression (Snappy, Zstd) and testing (GTest), managed via CMake and submodules. It emphasizes durability through Write-Ahead Logging (WAL) and robust recovery mechanisms that validate manifests and replay logs on startup. While the codebase demonstrates exceptional craftsmanship in concurrency control and data structures, it is receiving very limited maintenance, accepting only critical bug fixes. The repository requires C++17 and CMake 3.22, which may limit compatibility with older toolchains. The C API requires careful ABI management, and platform-specific implementations in `util/env_posix.cc` and `util/env_windows.cc` risk divergence over time.
This repository implements a high-performance distributed write-ahead log (WAL) system, written in Rust. It comprises a core WAL library (walrus-rust) with configurable storage backends (mmap, fd, io_uring), a Raft consensus library (openraft) for distributed metadata management, and a distributed application (distributed-walrus) that provides a TCP-based client-server architecture for topic-based streaming with lease-based write fencing and automatic segment rollover. The system is benchmarked against Kafka and RocksDB, includes a formal TLA+ specification, and is designed for crash-safe, low-latency persistent streaming. The architecture is layered: the Walrus WAL engine provides durable, topic-partitioned log storage with configurable fsync schedules and read consistency modes. The Octopii framework layer integrates QUIC transport and a flexible state machine. Openraft (vendored) provides full Raft consensus with leader election, log replication, snapshotting, and linearizable reads. The distributed-walrus application ties these together with a node controller, metadata state machine, and monitor loop for segment rollover. While primarily an application, the repository has strong research elements: extensive performance benchmarks (scaling, batch, multithreaded), formal verification via TLA+, and a focus on characterization of trade-offs between durability, latency, and throughput. The codebase is well-structured with clear abstractions, though it contains some duplicated code across the walrus and octopii directories and a large vendored copy of openraft.
SleekDB is a flat-file NoSQL database library implemented in PHP that stores data as plain JSON files. It provides a simple API for CRUD operations, querying with WHERE clauses, sorting, limiting, and caching. The architecture consists of a Store class as the main entry point, QueryBuilder for fluent query construction, and specialized helper classes for document finding, updating, and caching. Data is stored as individual JSON files in a data directory with a separate cache directory for query results. The library requires no third-party dependencies beyond PHP 8.4+ with JSON and MBString extensions. The codebase demonstrates solid engineering practices with proper file locking for concurrent access, custom exception hierarchy, and comprehensive test coverage using PHPUnit 11.0. The query builder follows the builder pattern with method chaining. Caching is implemented with automatic expiration based on configurable lifetimes. The library is designed for low to medium operation loads and is suitable for shared hosting environments. Some deprecated methods are marked for removal in version 3.0, indicating active maintenance and refactoring. The codebase maintains backward compatibility while preparing for future improvements.
Sophia is an embedded transactional key-value/row storage engine written in C, designed for high performance with ACID guarantees, MVCC (Multiversion Concurrency Control), and serializable snapshot isolation. Its architecture is inspired by LSM-trees but uses a unique node/branch/region model with O(1) worst-case set/delete and amortized O(branches) get. The system supports multiple databases, secondary indexes, custom comparators, upsert callbacks, compression (LZ4, Zstd), hot backup, online compaction/checkpointing, write-ahead logging (WAL), and crash recovery with fault injection. The codebase is modular with layers: a portable standard library (allocators, VFS, data structures, iterators, compression filters), a format/schema layer, an index layer, a transaction manager with deadlock detection, a scheduler for background jobs, and a public C API (sp_*). It includes an extensive test suite with unit, functional, crash, multithread, and hermitage isolation tests. The project is no longer actively developed, and the bundled third-party compression libraries are outdated, but the design and abstractions remain of educational and archival value.