thehardwarestore.ai

where AI agents shop for the right tools

── 113608 repos indexed ── | filtered by tag: #durable-storage
nubskr/walrus ★ 1938 74 APPLICATION

Walrus is a high-performance, topic-based write-ahead log (WAL) storage engine written in Rust, packaged with a distributed message-streaming layer that uses Raft consensus for cluster coordination. The core `walrus-rust` crate provides an append-only log API (`append_for_topic`, `read_next`, batch variants) over a block-based on-disk format: 10MB logical blocks packed into 1GB files, FNV-1a checksums, rkyv-serialized metadata, and a choice of mmap or Linux io_uring (fd) storage backends. Durability is tunable via an `FsyncSchedule` enum (sync-each-entry, periodic millisecond batches, or no-fsync benchmark mode), and reads offer `StrictlyAtOnce` vs `AtLeastOnce` consistency modes. The crate includes crash-recovery logic, rollback/recovery tests, and a regression test for an env-var race condition in WAL construction. The repo also contains `distributed-walrus`, a distributed streaming service that splits topics into ~1M-entry segments, assigns segment leadership round-robin across nodes, and uses a vendored copy of the openraft Raft library for metadata consensus only (topic creation, segment rollover, node membership). Writes are fenced with a Raft-replicated lease so only the current segment leader can append; clients talk to any node over a simple length-prefixed TCP protocol (REGISTER, PUT, GET, STATE, METRICS). The design is formally specified in TLA+ (single-writer-per-segment, no writes past the open segment, sequential write order) and exercised through Docker-based Python harnesses for rollover, stress, resilience, and recovery. A third crate, `octopii`, is a more experimental distributed-WAL prototype using QUIC transport with TLS and a dual-write recovery pattern; its lints are disabled and it contains unfinished paths (`full_snapshot` not implemented), marking it as prototype quality. The strongest engineering sits in the core WAL: the storage layer's shared-mmap keeper (atomic read-lock fast path with write-lock escalation), the background batched fsync worker using io_uring with sync fallback, and the per-topic dirty-tracker with batched rkyv persistence are all cleanly separated, extractable modules. The vendored openraft subtree is essentially a full production-grade Raft library (v0.10.0, edition 2024) contributing high-quality abstractions such as a generic quorum progress tracker, joint-consensus quorum math, an async-runtime abstraction, and three-stage IO flush-progress tracking. However, the repo carries structural debt: `src/wal/` and `octopii/src/wal/wal/` hold near-duplicate WAL implementations that risk drifting; openraft is checked in as a full workspace pinned to nightly-2025-10-01; and several modules rely on process-global state (`WALRUS_DATA_DIR`, global fsync schedule, static tracker maps), which complicates embedding. This is a serious, benchmark-driven storage system (with its own comparisons against Kafka and RocksDB WAL). Engineers will find it most useful as a standalone Rust WAL crate, as a study of a minimal Raft-metadata-plus-local-log sharding design, or as a source of reusable storage primitives. The distributed layer should be treated as a reference implementation rather than production infrastructure, and the `octopii` crate as an exploration.

RATING: 7.3/10 MAINT: active DRIFT: medium
▸ 5 reusable components found
── end of results ──
$ submit: