$ show NVIDIA/cudf
cuDF is a GPU-accelerated DataFrame library developed by NVIDIA as part of the RAPIDS ecosystem. It provides a pandas-compatible Python API (`cudf`), a Polars-compatible execution engine (`cudf_polars`), a zero-code-change pandas accelerator (`cudf.pandas`), distributed processing via Dask (`dask_cudf`), a streaming distributed execution engine (`cudf_streaming`), Kafka integration, Java bindings, and a low-level C++ library (`libcudf`). In production, cuDF accelerates data manipulation, ETL, analytics, and ML preprocessing on NVIDIA GPUs by executing columnar operations in CUDA kernels while managing GPU memory through RMM. The monorepo contains over 5000 files spanning C++, Python, Java, Rust, and shell, with extensive unit tests, benchmarks, CI pipelines, and documentation. Key architectural patterns include consistent stream/memory-resource propagation via `cudf::detail` namespaces, a result cache for aggregation reuse, a fast-slow proxy pattern for transparent GPU/CPU fallback, and a streaming actor model for distributed pipelines.
The LLM identified 6 files worth highlighting for reuse:
Implements a transparent proxy that delegates operations to either a fast GPU implementation or a slow CPU implementation based on availability. The proxy exposes the same interface as the underlying object and allows seamless fallback. This pattern is reusable for any library that needs to switch between accelerated and reference implementations at runtime, and is decoupled enough to extract with minimal changes.
Provides a cache for intermediate aggregation results, enabling reuse across multiple downstream operations without recomputing group keys or sorted orders. Implemented as a simple key-value store keyed by aggregation type and column pointer. This pattern can be extracted and applied to any iterative or multi-step aggregation pipeline to avoid redundant computation.
Implements the AllGather collective for distributed streaming execution using an actor graph. Includes an `Inserter` context manager for async insertion, memory reservation for concatenation costs, and thread-offloaded concatenation via `IRExecutionContext`. The pattern is reusable for any distributed data processing system that requires efficient multi-rank data exchange with bounded memory.
Generates binary operation methods (e.g., `__add__`, `__radd__`, `__iadd__`, `__divmod__`) for DataFrame and Series classes using a mixin factory. It automates the creation of reflected operators and fallback logic for unsupported dtypes. This metaprogramming pattern is reusable in any numerical computing library that needs to implement arithmetic operators across multiple types.
Tracks the provenance of column values through a query plan by maintaining a set of known values per column (column domain). Enables optimizations like predicate pushdown and avoiding unnecessary operations. The pattern is reusable in any query planner or optimizer that needs to reason about possible values at compile time.
Provides a robust library loader that first attempts to load a bundled shared library from a wheel, then falls back to the system-installed library. Uses `ctypes.CDLL` with `RTLD_LOCAL` to avoid symbol conflicts. This pattern is reusable for any Python package that distributes compiled libraries and needs to support both installed and bundled scenarios.