cuDF is a GPU-accelerated DataFrame library developed by NVIDIA as part of the RAPIDS ecosystem. It provides a pandas-compatible Python API (`cudf`), a Polars-compatible execution engine (`cudf_polars`), a zero-code-change pandas accelerator (`cudf.pandas`), distributed processing via Dask (`dask_cudf`), a streaming distributed execution engine (`cudf_streaming`), Kafka integration, Java bindings, and a low-level C++ library (`libcudf`). In production, cuDF accelerates data manipulation, ETL, analytics, and ML preprocessing on NVIDIA GPUs by executing columnar operations in CUDA kernels while managing GPU memory through RMM. The monorepo contains over 5000 files spanning C++, Python, Java, Rust, and shell, with extensive unit tests, benchmarks, CI pipelines, and documentation. Key architectural patterns include consistent stream/memory-resource propagation via `cudf::detail` namespaces, a result cache for aggregation reuse, a fast-slow proxy pattern for transparent GPU/CPU fallback, and a streaming actor model for distributed pipelines.
This is the NVIDIA RAPIDS Accelerator for Apache Spark (formerly cudf-spark), a high-performance Spark SQL plugin that replaces CPU-based Spark physical operators with GPU-accelerated equivalents using the cuDF library. The codebase provides a comprehensive shim layer supporting over 20 distinct Spark versions (3.3.x through 5.0.x) including Databricks runtimes, along with deep integrations for Delta Lake (2.x–4.x) and Apache Iceberg (1.6–1.11). It implements GPU versions of nearly every Spark SQL execution node: scans, joins, aggregates, windows, shuffles, file format readers/writers for Parquet, ORC, CSV, JSON, Avro, and Hive. The architecture relies on a sophisticated multi-version shim pattern, service-loading providers, and extensive use of Spark internal APIs to intercept and redirect execution to GPU columnar operations. The repository also includes a UDF compilation framework that converts Scala UDFs to GPU expressions, and a set of performance profiling and metrics tools. Despite its massive scope and high shim count, the codebase is well-structured with clear separation of concerns, comprehensive testing, and active release cadence. It is primarily an application (Spark plugin) with some auxiliary tooling for data generation and testing, but no data research elements.