Pathway is a high-performance live data processing framework built on top of timely dataflow and differential dataflow, providing a Python API for defining streaming and batch data pipelines. It is designed for real-time ETL, event processing, AI/ML pipelines (including RAG), and incremental computation with strong consistency guarantees. The architecture consists of a Rust core engine that handles dataflow execution, scheduling, persistence, and connectors, while the Python layer provides a declarative API for table operations, UDFs, and integrations with external systems like Kafka, databases, vector stores, and cloud storage. Key entry points include the `pathway` CLI, the Python `pw` module, and the Rust engine library. Data flows through connectors (sources/sinks), transformation operators (joins, groupby, windowing, etc.), and optional persistence for fault tolerance. External dependencies include Kafka, S3, PostgreSQL, various vector databases, and LLM services.
Apache Airflow is a workflow orchestration platform for authoring, scheduling, and monitoring data pipelines. The system uses a modular architecture with core components, extensible providers, a React/TypeScript web UI, and a FastAPI-based REST API. Key entry points include the CLI (`airflow` command), web server, scheduler, and triggerer. Data flows involve DAG parsing, task scheduling via executors, and state persistence in a metadata database. External dependencies include PostgreSQL/MySQL, Redis/RabbitMQ for Celery, and various provider-specific integrations. The codebase demonstrates strong engineering practices with version compatibility patterns, plugin systems, and well-structured API layers.
The Data Engineering Wiki is a community-maintained knowledge base dedicated to documenting data engineering concepts, tools, and best practices. Rather than being an executable application or data processing pipeline, it serves as a comprehensive reference resource for individuals learning and working in the data engineering field. The repository is organized into thematic sections covering foundational concepts (data architecture, modeling, processing, storage), practical tools and technologies, implementation guides, frequently asked questions, and curated learning resources. Structurally, the wiki follows a logical progression from theoretical foundations to practical application, with concepts explained through clear definitions, diagrams, and real-world examples. Key strengths include its comprehensive coverage of both established and emerging data engineering topics, practical guides on data pipeline best practices and testing methodologies, and active maintenance evidenced by recent commits. The content is presented in an accessible markdown format with consistent frontmatter, making it easy to navigate and consume. While not containing executable code or data pipelines itself, the wiki provides reusable knowledge assets in the form of methodological guides and reference materials that can be directly applied to data engineering projects. Notably, the Data Pipeline Best Practices and Testing Your Data Pipeline guides offer actionable frameworks for building reliable, scalable data systems. The repository's value lies in its role as a living document that captures and organizes community knowledge to support learning and professional practice in data engineering.
Flowcraft is a workflow orchestration framework for building, executing, and monitoring durable workflows. It provides a type-safe, composable API for defining workflows as directed graphs of nodes and edges, supporting both programmatic (fluent) and declarative (JSON) definitions. The framework enables progressive scaling from in-memory execution to distributed systems via adapters for message queues and cloud services, with built-in support for patterns like retries, timeouts, human-in-the-loop, batch processing, and subflows. Key architectural pieces include a dependency-free core runtime, a sophisticated context tracking system for efficient state persistence, a plugin-based adapter architecture for distributed execution, and LLM tooling for workflow introspection and control via natural language.
Feldera is a real-time streaming SQL analytics platform built on the DBSP (Differential Dataflow) incremental computation engine. It compiles SQL into efficient incremental circuits that process streaming data with sub-millisecond latency, supporting complex queries including joins, aggregations, window functions, and recursive queries. The system is deployed as a pipeline manager (Rust) with a REST API, a web console (Svelte), a Python SDK, a dbt adapter, and a CLI tool. It integrates with numerous data sources and sinks (Kafka, Postgres, Delta Lake, Iceberg, HTTP, NATS, Redis, etc.) and provides fault tolerance via checkpointing and storage backends. The architecture is polyglot: a Java-based SQL compiler (using Apache Calcite) translates SQL to Rust code that runs on the DBSP runtime; the pipeline manager orchestrates compilation, deployment, and lifecycle; the web console offers a full UI for pipeline management, monitoring, and profiling. The codebase also includes a Spark SQL translator (felderize) and extensive benchmark suites (Nexmark, LDBC). While primarily an application, it has strong research roots in incremental view maintenance and formal monotonicity analysis, reflected in the compiler's optimization passes and the DBSP runtime's operator design. Key reusable components include the adapter library (connector traits), the Feldera types crate (shared configs and errors), the common UI component library, and the Python client SDK. The project is actively maintained with frequent commits, comprehensive CI/CD, and a growing test suite covering SQL semantics across many data types and edge cases.
Exosphere is a reliability runtime for AI agents, providing a state-based workflow orchestration platform. The system consists of three main components: a Python SDK (`exospherehost`) for defining nodes and execution graphs, a FastAPI state-manager backend that handles state transitions, retries, cron triggers, and secret encryption, and a Next.js dashboard for monitoring and managing workflows. The architecture follows a microservices-like pattern within a monorepo, with the SDK acting as a client library and the state-manager as the central orchestration engine. Key capabilities include dynamic graph execution, infinite parallel fanout/unite patterns, configurable retry policies (exponential, linear, fixed with jitter), signal-based state pruning and re-queuing, and a graph-level key-value store. The system relies on MongoDB for persistence and uses Beanie ODM. The codebase demonstrates solid engineering practices with well-structured controllers, middleware for request ID propagation and exception handling, and a comprehensive test suite, though some tests are incomplete or skipped. The project is in early beta (v0.0.3b1) with active development and monthly releases.
Mage is an open-source data pipeline orchestration platform that combines a visual notebook-style editor with a production-grade scheduling and execution engine. The system enables users to build, run, and monitor data pipelines composed of Python, SQL, R, and dbt blocks. Architecturally, it is a full-stack web application with a Python backend (Tornado-based server, SQLAlchemy ORM, REST API) and a Next.js/React frontend with a custom component library. Key entry points include the CLI (`mage` command), the web UI served by `mage_ai/server/server.py`, and the API layer at `mage_ai/api/` which exposes CRUD resources for pipelines, blocks, runs, schedules, users, and integrations. Data flows center around pipeline execution: blocks are connected in a DAG, each block can be a data loader, transformer, or exporter. Execution is handled by a configurable executor (local, K8s, EMR, PySpark) and scheduled via a built-in scheduler or external triggers. The platform integrates with dozens of data sources and destinations through a Singer-based integration framework, supports streaming pipelines, and includes advanced features like RBAC, version control, AI-assisted pipeline generation, and a command center for global search. It depends heavily on external services such as cloud storage (S3, GCS, Azure), databases (PostgreSQL, BigQuery, Snowflake, etc.), and authentication providers (OAuth, LDAP, OIDC).