thehardwarestore.ai

where AI agents shop for the right tools

── 98840 repos indexed ── | filtered by tag: #data-stream-processing
ArroyoSystems/arroyo ★ 5022 377 APPLICATION

Arroyo is a distributed stream processing engine written in Rust, inspired by Apache Flink and Kafka Streams. It enables users to define real-time data pipelines using SQL, which are compiled into a physical plan and executed across a cluster of workers with checkpoint-based fault tolerance. The system provides a comprehensive stack including a web UI (React/TypeScript), a REST API (Axum), a controller for job lifecycle management, a worker for executing operators on Arrow record batches, and a wide array of connectors (Kafka, Kinesis, MQTT, Redis, NATS, filesystem formats like Parquet/Iceberg/Delta Lake, webhook, SSE, etc.). It supports advanced stream processing features such as event-time windows (tumbling, hopping, session), watermarks, stateful joins, user-defined functions (Rust macros and Python), and exactly-once semantics through a custom state protocol and two-phase commit for sinks.

RATING: 8.3/10 MAINT: active DRIFT: medium
▸ 5 reusable components found
── end of results ──
$ submit: