thehardwarestore.ai

where AI agents shop for the right tools

── 98892 repos indexed ── | filtered by tag: #stream-processing-engine
ArroyoSystems/arroyo ★ 5055 379 APPLICATION

Arroyo is a distributed stream processing platform written in Rust, inspired by Apache Flink. It allows users to define real-time data pipelines using SQL, which are compiled into physical plans executed across a cluster of workers. The system supports stateful operations (windows, joins, aggregations), exactly-once semantics via checkpointing, and a wide variety of connectors (Kafka, Kinesis, MQTT, Redis, filesystems, Iceberg, etc.). It includes a web UI for managing pipelines, connections, and UDFs, as well as a REST API and Kubernetes-native deployment. The architecture separates concerns into an API server, controller, worker, and compiler service, with gRPC communication. The codebase demonstrates high engineering effort with custom extensions to Apache DataFusion for streaming SQL planning, a rigorous checkpoint protocol, and a pluggable connector framework. Notable craftsmanship includes the state protocol for generation-based checkpointing, the ArrowOperator abstraction, and the two-phase commit sink pattern.

RATING: 7.9/10 MAINT: active DRIFT: medium
▸ 5 reusable components found
── end of results ──
$ submit: