Delta Lake is a transactional table format and multi-engine connector suite. In production it writes Parquet data files alongside a JSON transaction log (`_delta_log`) and periodic checkpoints; every commit is an atomic append of log actions. Readers reconstruct table snapshots by replaying the log and checkpoints, enabling ACID operations, time travel, schema evolution, and incremental streaming on object storage. This repository is the reference implementation of that protocol, not an end-user data platform. The architecture is a multi-module Scala/Java/Python monorepo: `spark/` contains the Spark V1 and V2 connectors plus SQL extensions; `kernel/` is a standalone Java, engine-agnostic library exposing `Engine`, `TableManager`, `TransactionBuilder`, and a default Hadoop engine; `storage/` provides LogStore abstractions and cloud-specific commit coordination; and `flink/`, `iceberg/`, `sharing/`, and `python/delta` provide additional engine integrations. Key entry points are `DeltaTable`, `DeltaSparkSessionExtension`, `io.delta.kernel.TableManager`, and Flink's `DeltaDynamicTableSource`/`DeltaDynamicTableSink`. The central data flow is client -> `OptimisticTransaction`/Kernel `Transaction` -> log action -> LogStore commit; reads use `Snapshot`/`SnapshotBuilder` to replay log actions. External dependencies include Apache Spark, Hadoop FileSystem, Parquet, Jackson, and optional cloud SDKs. Maintenance is genuinely active: recent commits, CI matrices for Spark 4.0-4.2, Flink, Iceberg, Kernel, and Unity Catalog, plus a large compatibility test suite. The codebase nevertheless carries transitional complexity: a V2/Kernel-backed Spark connector coexists with the legacy V1 path, many Spark-version shims exist, test suites are partly auto-generated, and the old `connectors/` directory is explicitly in maintenance mode. Those factors prevent a perfect score, but this is a mature, high-craft storage-format project.
redb is a high-performance, ACID-compliant embedded key-value store library written in Rust. It provides a B-tree-based storage engine with copy-on-write semantics, MVCC concurrency control, and support for typed tables, multimaps, savepoints (ephemeral and persistent), compaction, and crash recovery. The library exposes a clean public API centered around `Database`, `TableDefinition`, `WriteTransaction`, `ReadTransaction`, and `ReadableTable` traits, with optional derive macros for user-defined key/value types. It is designed for single-writer, multiple-reader access and supports multi-process concurrency via byte-range file locking (experimental). The codebase is extensively tested with integration tests, fuzzing, and benchmarks comparing against LMDB, RocksDB, SQLite, and fjall. Architecturally, redb uses a copy-on-write B+tree for all tables (data, system, freed pages) with a buddy allocator for page management and a region-based file layout. The page store includes a striped LRU read cache, write buffer, and dynamic cache partitioning. Commit strategies range from non-durable (no fsync) to 1PC+C (default) and 2PC (secure). Crash recovery relies on XXH3-128 checksums and optional quick-repair via persistent allocator state. The library is `no_std`-compatible with feature-gated std-dependent parts, and includes a custom spin-lock synchronization module for embedded targets. Key entry points include the `Database::builder()` and `Database::open()` methods, `TableDefinition::new()` for schema definition, and transaction methods like `WriteTransaction::commit()` and `ReadTransaction::open_table()`. The `StorageBackend` trait allows pluggable storage (file, in-memory, custom) and is used extensively in tests for fault injection and crash simulation. External dependencies are minimal (standard library, optional chrono/uuid), with benchmark crates depending on heed, rocksdb, fjall, and rusqlite for comparison. The repository demonstrates strong engineering craftsmanship with clean abstractions, careful handling of edge cases (max keys, large values, corruption), and a well-documented design document (`docs/design.md`). The codebase is actively maintained with a stable file format and backward compatibility guarantees.
Kronotop is a distributed database system built on FoundationDB that provides Redis-compatible (RESP2/RESP3) interfaces for multiple data models including key-value (stash), document (bucket with BSON storage and querying), and sorted sets (zmap). It features ACID transactions, secondary indexing (including vector search via HNSW), clustering, sharding, replication, and namespace isolation for multi-tenancy. The architecture is modular with clear separation between network handling (Netty), protocol processing, service layers (transaction, session, data services), and storage (volume-based segment store). Key entry points include the server entry point (KronotopInstanceStarter), CLI (kronotop-cli), and control tool (kronotop-ctl). The system relies on external dependencies such as FoundationDB, Netty, Jackson, Lettuce, Guava, JVector, and Typesafe Config. The codebase demonstrates strong engineering practices with reusable components like versionstamp utilities, retry mechanisms, and protocol handling.
Sophia is an embedded transactional key-value/row storage engine written in C, designed for high performance with ACID guarantees, MVCC (Multiversion Concurrency Control), and serializable snapshot isolation. Its architecture is inspired by LSM-trees but uses a unique node/branch/region model with O(1) worst-case set/delete and amortized O(branches) get. The system supports multiple databases, secondary indexes, custom comparators, upsert callbacks, compression (LZ4, Zstd), hot backup, online compaction/checkpointing, write-ahead logging (WAL), and crash recovery with fault injection. The codebase is modular with layers: a portable standard library (allocators, VFS, data structures, iterators, compression filters), a format/schema layer, an index layer, a transaction manager with deadlock detection, a scheduler for background jobs, and a public C API (sp_*). It includes an extensive test suite with unit, functional, crash, multithread, and hermitage isolation tests. The project is no longer actively developed, and the bundled third-party compression libraries are outdated, but the design and abstractions remain of educational and archival value.