$ show nubskr/walrus
This repository implements a high-performance distributed write-ahead log (WAL) system, written in Rust. It comprises a core WAL library (walrus-rust) with configurable storage backends (mmap, fd, io_uring), a Raft consensus library (openraft) for distributed metadata management, and a distributed application (distributed-walrus) that provides a TCP-based client-server architecture for topic-based streaming with lease-based write fencing and automatic segment rollover. The system is benchmarked against Kafka and RocksDB, includes a formal TLA+ specification, and is designed for crash-safe, low-latency persistent streaming. The architecture is layered: the Walrus WAL engine provides durable, topic-partitioned log storage with configurable fsync schedules and read consistency modes. The Octopii framework layer integrates QUIC transport and a flexible state machine. Openraft (vendored) provides full Raft consensus with leader election, log replication, snapshotting, and linearizable reads. The distributed-walrus application ties these together with a node controller, metadata state machine, and monitor loop for segment rollover. While primarily an application, the repository has strong research elements: extensive performance benchmarks (scaling, batch, multithreaded), formal verification via TLA+, and a focus on characterization of trade-offs between durability, latency, and throughput. The codebase is well-structured with clear abstractions, though it contains some duplicated code across the walrus and octopii directories and a large vendored copy of openraft.
The LLM identified 4 files worth highlighting for reuse:
A builder for constructing `Walrus` WAL instances with explicit parameters (data directory, namespace key, read consistency, fsync schedule, block size, file size). This pattern avoids the project's earlier use of global environment variables and race conditions, and is cleanly extractable for any project needing a durable, configurable log storage instance.
A generic, efficient quorum progress tracker that maintains a sorted vector of progress values and computes the quorum-accepted value in O(n log n). It is used in Raft to track replication progress across followers, but the pattern is reusable for any distributed protocol that needs to determine a threshold value (e.g., commit index, clock sync).
An abstract `AsyncRuntime` trait that defines spawn, sleep, timeout, channel, mutex, oneshot, and watch operations. This enables openraft to work with Tokio, monoio, or compio runtimes without modification. The pattern is a clean separation of runtime concerns, reusable by any async library that needs runtime portability.
An enum-based storage backend (`Mmap` or `Fd`) with a shared `MmapKeeper` that uses `OnceLock<RwLock<HashMap>>` to ensure single mapping per file path. This pattern provides a thread-safe, reusable storage abstraction for WAL or other persistent data structures, supporting both memory-mapped and direct I/O backends with double-checked locking for initialization.