$ show feldera/feldera
Feldera is a real-time streaming SQL analytics platform built on the DBSP (Differential Dataflow) incremental computation engine. It compiles SQL into efficient incremental circuits that process streaming data with sub-millisecond latency, supporting complex queries including joins, aggregations, window functions, and recursive queries. The system is deployed as a pipeline manager (Rust) with a REST API, a web console (Svelte), a Python SDK, a dbt adapter, and a CLI tool. It integrates with numerous data sources and sinks (Kafka, Postgres, Delta Lake, Iceberg, HTTP, NATS, Redis, etc.) and provides fault tolerance via checkpointing and storage backends. The architecture is polyglot: a Java-based SQL compiler (using Apache Calcite) translates SQL to Rust code that runs on the DBSP runtime; the pipeline manager orchestrates compilation, deployment, and lifecycle; the web console offers a full UI for pipeline management, monitoring, and profiling. The codebase also includes a Spark SQL translator (felderize) and extensive benchmark suites (Nexmark, LDBC). While primarily an application, it has strong research roots in incremental view maintenance and formal monotonicity analysis, reflected in the compiler's optimization passes and the DBSP runtime's operator design. Key reusable components include the adapter library (connector traits), the Feldera types crate (shared configs and errors), the common UI component library, and the Python client SDK. The project is actively maintained with frequent commits, comprehensive CI/CD, and a growing test suite covering SQL semantics across many data types and edge cases.
The LLM identified 5 files worth highlighting for reuse:
Implements a dynamic-typed `Pairs` trait for key-value pair operations over `LeanVec`, with methods for sorting, deduplication (keeping last value per key), and trait object dispatch via `DynPairs`. This is a core abstraction used throughout the DBSP runtime for handling generic pair data without static types, demonstrating advanced Rust patterns for runtime polymorphism and efficient vector manipulation.
Defines the base traits and types for input/output adapters in Feldera, including `InputEndpoint`, `OutputEndpoint`, `TransportConfig`, and format handling. This library is self-contained and could be extracted to build custom connectors for other streaming systems. It demonstrates clean trait design with minimal internal coupling.
A reusable Svelte 5 component wrapping the Monaco editor with rune-based reactivity (`$state`, `$effect`). It provides a clean interface for embedding a code editor with customizable language, theme, and content binding. Demonstrates modern Svelte patterns and can be reused in any Svelte project needing a code editor.
Central crate defining shared types for the Feldera platform: pipeline configs, transport configs, error types, checkpoint metadata, and format specifications. It uses serde for serialization and is the single source of truth for data structures exchanged between the pipeline manager, adapters, and runtime. Highly reusable as a schema library for any system interacting with Feldera.
A reusable utility for tracking which fields of a record are used in an expression or circuit. Used by multiple optimization passes (e.g., `TrimWindows`, `UnusedFields`) to eliminate unnecessary computations. Demonstrates a clean, composable design for field-level dependency analysis that could be applied to other query compilers.