$ show pola-rs/polars
Polars is a high-performance DataFrame library and query engine written in Rust with Python bindings. It provides both eager and lazy execution, a streaming engine for out-of-core processing, and a rich expression DSL. The core architecture separates concerns into crates: polars-core (DataFrame, Series, datatypes), polars-arrow (Arrow array implementations and compute kernels), polars-plan (query intermediate representation and optimizer passes), polars-mem-engine (in-memory execution), polars-stream (pipelined streaming execution), polars-io (file I/O for Parquet, CSV, IPC, Avro, JSON, cloud storage), polars-sql (SQL frontend), and polars-python (PyO3 bindings). Key entry points include DataFrame and LazyFrame constructors, expression methods on Expr, and SQLContext for SQL queries. Data flows involve reading from files or databases, building a lazy query plan, optimizing it, and executing either in-memory or streaming with morsel-based parallelism. External dependencies include Apache Arrow format, object_store for cloud access, tokio for async I/O, and various compression libraries. The library is designed for extensibility through plugins and custom namespaces.
The LLM identified 5 files worth highlighting for reuse:
Defines the `RollingAggWindowNoNulls` and `RollingAggWindowNulls` traits for composing rolling window aggregates (sum, mean, min, max, variance, skew, kurtosis). Each kernel implements a sliding window computation with optional null handling. The pattern is reusable for any columnar library needing efficient rolling operations, as it abstracts the window iteration logic from the aggregation function.
Implements encoding of Arrow arrays into byte-comparable row keys for sorting and comparison. Supports fixed-width types (booleans with sentinel values, numerics, decimals) and variable-width types (binary, utf8, no-order). The encoding ensures that byte comparison yields the same ordering as the original data. This is a reusable pattern for any system needing efficient sorting or hashing of heterogeneous rows.
Provides an `Arena<T>` for storing nodes (e.g., expression trees, query plan nodes) with `Node` handles. Supports cloning, iteration, and safe mutation. This pattern is critical for performance in query optimizers and is reusable for any graph-heavy computation that requires stable references without lifetimes.
Contains the `LowerIr` trait and its implementation that translates the optimized logical IR into a physical streaming pipeline of `ComputeNode` instances. This demonstrates a clean separation between logical optimization and physical execution, with reusable visitor patterns and type dispatch for building node graphs.
Implements the `StaticArrayBuilder` trait and concrete builders for primitive, boolean, binary, and nested arrays. Provides a uniform interface for constructing Arrow arrays with `reserve`, `push`, `extend`, `freeze`, and `reset`. The trait is generic and can be used by any columnar processing library that needs to build Arrow arrays incrementally.