Pathway is a high-performance live data processing framework built on top of timely dataflow and differential dataflow, providing a Python API for defining streaming and batch data pipelines. It is designed for real-time ETL, event processing, AI/ML pipelines (including RAG), and incremental computation with strong consistency guarantees. The architecture consists of a Rust core engine that handles dataflow execution, scheduling, persistence, and connectors, while the Python layer provides a declarative API for table operations, UDFs, and integrations with external systems like Kafka, databases, vector stores, and cloud storage. Key entry points include the `pathway` CLI, the Python `pw` module, and the Rust engine library. Data flows through connectors (sources/sinks), transformation operators (joins, groupby, windowing, etc.), and optional persistence for fault tolerance. External dependencies include Kafka, S3, PostgreSQL, various vector databases, and LLM services.
This is a comprehensive visual programming system called 'Unit' that implements a dataflow/reactive paradigm where programs are composed of interconnected units (MIMO finite state machines) in a directed graph. The system includes a full-stack runtime (Node.js server and browser client), a graph editor UI with drag-and-drop, a rich library of built-in functional units (arithmetic, string, array, control flow, etc.), platform API wrappers (canvas, audio, Bluetooth, WebSocket, etc.), and a custom spec language for defining units. The architecture is centered around a core class hierarchy (Unit, Graph, Pin, Component) with a reactive push-based execution model. The project is actively maintained with a large community (4920 stars) and demonstrates sophisticated engineering in areas like force-directed layout simulation, custom parser with type inference, and a comprehensive debugging infrastructure. While the codebase is extensive and well-structured, some components (e.g., the main Component class) are large and tightly coupled, and there are signs of duplication and incomplete implementations in some areas.
Arroyo is a distributed stream processing platform written in Rust, inspired by Apache Flink. It allows users to define real-time data pipelines using SQL, which are compiled into physical plans executed across a cluster of workers. The system supports stateful operations (windows, joins, aggregations), exactly-once semantics via checkpointing, and a wide variety of connectors (Kafka, Kinesis, MQTT, Redis, filesystems, Iceberg, etc.). It includes a web UI for managing pipelines, connections, and UDFs, as well as a REST API and Kubernetes-native deployment. The architecture separates concerns into an API server, controller, worker, and compiler service, with gRPC communication. The codebase demonstrates high engineering effort with custom extensions to Apache DataFusion for streaming SQL planning, a rigorous checkpoint protocol, and a pluggable connector framework. Notable craftsmanship includes the state protocol for generation-based checkpointing, the ArrowOperator abstraction, and the two-phase commit sink pattern.
Etl-engine is a Java-based ETL (Extract, Transform, Load) library designed for high-performance data synchronization. It provides a lightweight, extensible framework built around three core components: Node (data sources/destinations/transformations), Pipe (thread-safe buffered queues for data transfer), and Dataflow (orchestrator for pipeline execution). The library supports various data sources including JDBC databases (PostgreSQL, MySQL, Oracle, ClickHouse, DuckDB) and file outputs, with built-in capabilities for batch processing, automatic error handling/retry, and detailed logging. The architecture follows a producer-consumer pattern using Java's concurrency utilities, enabling efficient parallel data processing. Sample implementations demonstrate common ETL scenarios like database-to-database transfers, conditional data routing, and value transformation.