$ show apache/fluss
Apache Fluss is a distributed streaming storage system optimized for real-time analytics and AI workloads. It provides both log (append-only) and key-value (upsert) table semantics, with sub-second freshness and support for columnar formats via Apache Arrow. The system follows a two-tier server architecture: a coordinator server managing metadata, rebalancing, and ACLs, and tablet servers handling data storage, replication, and snapshot management. It integrates deeply with Apache Flink (source, sink, catalog, tiering) and Apache Spark (catalog, read/write), and includes lake tiering to Iceberg, Hudi, Paimon, and Lance formats. A native Rust client with Python, Elixir, and C++ bindings provides multi-language access, and a stateless REST gateway is available. The codebase demonstrates strong engineering practices: a clean SPI-based plugin system for filesystems, metric reporters, authentication, and lake storage; a comprehensive exception hierarchy; thorough metrics and monitoring; full-text predicate pushdown; and sophisticated memory management with off-heap memory segments and paged output views. Configuration is modular with providers for files, directories, and environment variables. The project includes extensive unit, integration, and behavior tests for filesystems, replication, rebalancing, and Flink compatibility across multiple versions. Notable reusable components include the throwing functional interfaces for checked exceptions, a clean JSON serialization framework with versioned test patterns, and a memory management subsystem offering reusable memory segment pools. The system is production-ready, with active development, low known security alerts, and continuous integration across multiple OS and storage backends.
The LLM identified 3 files worth highlighting for reuse:
A set of functional interfaces (ThrowingConsumer, ThrowingSupplier, etc.) that mirror java.util.function but allow checked exceptions. They include a static 'unchecked' method that wraps a throwing interface into a standard one by rethrowing exceptions as RuntimeException. This is a clean, self-contained pattern for using lambda expressions with checked exceptions in streams and other functional APIs, common to many Java projects.
Provides a simple, generic JsonSerializer/JsonDeserializer interface pair coupled with JsonSerdeUtils for object mapping. The framework supports versioned data (e.g., metadata stored in ZooKeeper) and includes a reusable JsonSerdeTestBase for testing round-trip serialization. This is a clean, extensible pattern that can be extracted for any project needing JSON persistence with versioning and testability.
Implements a fixed-size memory buffer (heap or direct) with efficient read/write operations using sun.misc.Unsafe. Combined with AbstractPagedOutputView, it provides a paged, sequential writing mechanism ideal for batch record serialization. The design is carefully optimized for zero-copy and low garbage, and includes a MemorySegmentPool for reusing segments. This component, though dependent on Fluss internals for Unsafe access, demonstrates a sophisticated memory management pattern commonly needed in high-performance data systems.