$ show superfly/litefs
LiteFS is a FUSE-based distributed filesystem for replicating SQLite databases across a cluster of machines. It operates as a Linux daemon that intercepts SQLite's file-level operations (journal, WAL, SHM) to capture every transaction. These transactions are recorded in a custom LTX (Lite Transaction) binary format and streamed to replica nodes via an HTTP/2 h2c protocol. The system uses leader election (via Consul or a static configuration) to designate a primary node that accepts writes, while replicas synchronously apply changes and enforce read-after-write consistency via TXID cookies and a proxy layer. The architecture is fundamentally a thin FUSE layer wrapping a distributed state machine: each node runs a store that manages multiple databases, tracks guards for SQLite lock compatibility (PENDING, SHARED, etc.), and exposes an HTTP API for streaming replication, manual imports/exports, handoff, and backup integration. The proxy server sits in front of the application to route read requests to the primary or a sufficiently caught-up replica, using Fly-Replay headers for transparent redirect. Backup is supported through either a file-based mechanism or the LiteFS Cloud client. External dependencies include Consul (for leader election), FUSE (required on Linux), and the underlying SQLite library (no cgo, pure Go via the ltxx package). The codebase is written in Go, with a heavy focus on correctness under concurrency—evidenced by extensive integration tests that exercise split-brain recovery, checkpoint races, and multi-node handoff. The project is in beta and follows a roadmap-driven contribution model where the maintainer rewrites feature PRs to ensure consistency.
The LLM identified 3 files worth highlighting for reuse:
Provides a generic Reader and Writer for chunked byte streams with a maximum chunk size (64KB). The writer writes data in fixed-sized chunks with an EOF marker, while the reader reassembles them. This is an isolated, stdlib-only utility that can be extracted and reused in any Go project needing streaming over unbounded or limited transports (e.g., HTTP, files). It demonstrates clean error handling and idempotent read semantics.
Implements a custom RWMutex that supports lock upgrading (shared to exclusive) and downgrading (exclusive to shared) with state-change callbacks. It is designed to mimic SQLite's lock protocol (PENDING, SHARED, RESERVED, WRITE) and integrates with the project's GuardSet. The implementation is self-contained (depends only on standard library synchronization primitives) and can be reused in any project requiring advanced read/write lock semantics beyond Go's built-in RWMutex.
A collection of small, opinionated utility functions for file system operations: Sync (directory fsync), ReadFullAt (retry on short reads), and Close (silently ignores benign errors like network closes or already-closed files). Each function solves a recurring distributed-systems problem in a safe, idempotent way. These are reusable across any Go project that interfaces with files or sockets, and they avoid importing any project-internal packages.