Apache Polaris is a production-grade, multi-tenant Iceberg REST catalog server built on Quarkus. It provides a REST API for catalog management (namespaces, tables, views), credential vending for cloud storage (S3, GCS, Azure), fine-grained authorization via built-in RBAC or external OPA/Ranger, and event publishing (Kafka, CloudWatch). The system supports two persistence backends: a legacy relational JDBC (PostgreSQL, H2) and a new NoSQL abstraction with custom indexing, caching, and distributed invalidation. Multi-tenancy is implemented via isolated realms, each with its own configuration and persisted state. Key entry points are the Quarkus server (`polaris-server`), a CLI admin tool, and a Python client. External dependencies include Iceberg libraries, Quarkus, Jackson, Testcontainers, and optional integrations like Open Policy Agent, Apache Ranger, and Kafka. The architecture is modular with clear SPI boundaries, making extensions (persistence, auth, federation) pluggable.
Apache Iceberg is a high-performance table format for analytic data lakes, providing ACID transactions, snapshot isolation, schema evolution, and time travel. The repository contains the Java reference implementation, including core libraries for table metadata management, file format support (Parquet, ORC, Avro, Arrow), expression evaluation, encryption, and a REST catalog protocol. It integrates deeply with Spark (versions 3.5, 4.0, 4.1, 4.2), Flink (1.20, 2.2, 2.3), Kafka Connect, Hive, and cloud object stores (AWS S3, Azure ADLS, Aliyun OSS, GCS). The architecture is modular: the `api` module defines public interfaces (Table, Catalog, Scan, FileIO), `core` provides implementations and utilities, and separate modules handle engine-specific integrations and cloud backends. Key entry points include the Table interface for DDL/DML operations, Scan for query planning, FileIO for storage access, and the REST catalog client for remote metadata management. Data flows involve reading/writing manifest files, applying expressions for predicate pushdown, and coordinating commits via optimistic concurrency control. External dependencies include Apache Parquet, ORC, Avro, Arrow, and various cloud SDKs, but the core is designed to be lightweight and extensible.
This is the NVIDIA RAPIDS Accelerator for Apache Spark (formerly cudf-spark), a high-performance Spark SQL plugin that replaces CPU-based Spark physical operators with GPU-accelerated equivalents using the cuDF library. The codebase provides a comprehensive shim layer supporting over 20 distinct Spark versions (3.3.x through 5.0.x) including Databricks runtimes, along with deep integrations for Delta Lake (2.x–4.x) and Apache Iceberg (1.6–1.11). It implements GPU versions of nearly every Spark SQL execution node: scans, joins, aggregates, windows, shuffles, file format readers/writers for Parquet, ORC, CSV, JSON, Avro, and Hive. The architecture relies on a sophisticated multi-version shim pattern, service-loading providers, and extensive use of Spark internal APIs to intercept and redirect execution to GPU columnar operations. The repository also includes a UDF compilation framework that converts Scala UDFs to GPU expressions, and a set of performance profiling and metrics tools. Despite its massive scope and high shim count, the codebase is well-structured with clear separation of concerns, comprehensive testing, and active release cadence. It is primarily an application (Spark plugin) with some auxiliary tooling for data generation and testing, but no data research elements.
This repository implements a Cloudflare Worker that proxies HTTP requests to a Durable Object-backed Container running a DuckDB API. The Worker forwards environment variables to enable optional features: API token authentication, R2 Data Catalog for Iceberg support, and Postgres+R2 for DuckLake integration. The Container initializes DuckDB lazily on first request, loads extensions (httpfs, avro, iceberg, ducklake, postgres_scanner, spatial, nanoarrow), and conditionally attaches remote catalogs based on available secrets. The API provides endpoints for schema inspection and SQL querying, with query filtering to block dangerous operations. The architecture separates concerns: Worker handles routing and secret forwarding, Container manages the Hono server and DuckDB lifecycle, and lib modules encapsulate database utilities, logging, and query safety.
pg_mooncake is a Postgres extension that adds a custom table access method ('mooncake') backed by Apache Iceberg, enabling real-time analytics on columnar data stored in Iceberg tables. It integrates DuckDB (via pg_duckdb) as the query engine for fast analytics and provides a background worker (moonlink) that streams changes from regular Postgres heap tables to Iceberg tables via logical replication, ensuring sub-second data freshness. The extension exposes SQL procedures to create columnstore mirrors of Postgres tables, manage snapshots, load files, and optimize storage. It is built using the pgrx framework for Rust-based Postgres extensions and supports Postgres versions 14-18. The architecture is a hybrid of a shared library loaded into the Postgres backend and a separate background worker process. The codebase is well-structured and actively developed, though many table access method callbacks are still stubs, indicating it is in an early but functional stage.