$ show ankane/ruby-polars
This repository is a Ruby gem (polars-df) that provides a high-performance DataFrame library by binding to the Rust-based Polars engine. It serves as a native extension, allowing Ruby developers to perform ETL, data transformation, and analytics tasks with Rust-level speed while maintaining Ruby syntax. The architecture consists of a Rust core (ext/polars) handling computation and memory management, wrapped by Ruby modules (lib/polars) that expose a fluent DSL for data manipulation. It supports both eager (DataFrame/Series) and lazy (LazyFrame/Expr) evaluation modes, with extensive I/O capabilities for formats like CSV, Parquet, and Arrow, as well as cloud storage and database integration. The codebase demonstrates strong engineering practices, utilizing the `magnus` crate for safe Ruby-Rust FFI and implementing complex type conversion logic to bridge the two languages. It includes comprehensive test suites and CI pipelines for cross-platform compilation. However, the tight coupling to the upstream Polars Rust library introduces maintenance overhead, as breaking changes in the Rust core require synchronous updates to the Ruby bindings. Several features, particularly around cloud catalogs (Iceberg, Delta) and advanced selectors, are marked as experimental or contain incomplete implementations (Todo blocks), indicating an active but evolving codebase. For AI agents and engineers, this library is the primary tool for data engineering within the Ruby ecosystem where performance is critical. It is suitable for building data pipelines, analytics services, or backend systems that require efficient data processing without leaving the Ruby environment. The presence of `unstable` markers and `Todo` blocks suggests that while the core functionality is robust, some advanced features should be used with caution in production environments.
The LLM identified 3 files worth highlighting for reuse:
Implements the critical type mapping logic between Ruby objects and Rust Polars types. This module handles the conversion of complex data structures (categorical, list, struct) across the FFI boundary, ensuring memory safety and correct data representation. It demonstrates non-obvious skill in managing the Ruby GC and Rust lifetimes simultaneously, making it a reusable pattern for any Ruby-Rust binding project requiring complex type interoperability.
Defines the `Polars::Selectors` (cs) module, providing a fluent API for column selection based on dtype, name patterns, or set operations. This component encapsulates a reusable pattern for declarative column targeting, allowing users to write `cs.numeric()` or `cs.matches('date.*')` instead of hardcoding column names. It demonstrates clean abstraction and method chaining, making it easily extractable for similar column-selection DSLs in other Ruby libraries.
Handles the normalization of diverse input types (Hash, Array, Series, Arrow C Stream) into a consistent internal representation before passing data to the Rust backend. This module implements a factory-like pattern that validates schema and data types, preventing downstream errors in the Rust layer. It is a reusable component for any data ingestion pipeline that needs to support multiple input formats while maintaining strict type safety.