$ show ArroyoSystems/arroyo
Arroyo is a distributed stream processing platform written in Rust, inspired by Apache Flink. It allows users to define real-time data pipelines using SQL, which are compiled into physical plans executed across a cluster of workers. The system supports stateful operations (windows, joins, aggregations), exactly-once semantics via checkpointing, and a wide variety of connectors (Kafka, Kinesis, MQTT, Redis, filesystems, Iceberg, etc.). It includes a web UI for managing pipelines, connections, and UDFs, as well as a REST API and Kubernetes-native deployment. The architecture separates concerns into an API server, controller, worker, and compiler service, with gRPC communication. The codebase demonstrates high engineering effort with custom extensions to Apache DataFusion for streaming SQL planning, a rigorous checkpoint protocol, and a pluggable connector framework. Notable craftsmanship includes the state protocol for generation-based checkpointing, the ArrowOperator abstraction, and the two-phase commit sink pattern.
The LLM identified 5 files worth highlighting for reuse:
Implements a rigorous checkpoint and commit protocol using generations, epochs, and committed markers. Provides functions for initializing generations, publishing checkpoints, authorizing commits, and detecting orphaned checkpoints. This protocol could be extracted and reused in any distributed system requiring exactly-once semantics with fault tolerance.
Defines the Connector trait and ErasedConnector abstraction, providing a uniform interface for implementing sources and sinks. Connectors are registered with metadata, testing, and operator construction methods. This pattern allows pluggable data sources/sinks with minimal effort, and could be reused in other stream-processing engines.
Defines the TwoPhaseCommitter trait for implementing exactly-once file system sinks, with strategies for per-subtask and per-operator committing. The pattern decouples commit logic from the sink operator, enabling reuse across different storage backends (local, S3, Iceberg, Delta).
A generic React component that renders a form from a JSON Schema, supporting oneOf, arrays, maps, nested objects, autocomplete, and validation. It is used for configuring connectors in the UI but is decoupled from any specific domain logic, making it reusable in any project requiring schema-driven form generation.
Contains the ArrowOperator and SourceOperator traits that define a common lifecycle for all processing operators (on_start, process_batch, handle_checkpoint, etc.). These traits are used by all connectors and transforms in Arroyo but are self-contained and could serve as a template for building streaming operators in other Rust projects.