$ show apache/iceberg
Apache Iceberg is a high-performance table format for analytic data lakes, providing ACID transactions, snapshot isolation, schema evolution, and time travel. The repository contains the Java reference implementation, including core libraries for table metadata management, file format support (Parquet, ORC, Avro, Arrow), expression evaluation, encryption, and a REST catalog protocol. It integrates deeply with Spark (versions 3.5, 4.0, 4.1, 4.2), Flink (1.20, 2.2, 2.3), Kafka Connect, Hive, and cloud object stores (AWS S3, Azure ADLS, Aliyun OSS, GCS). The architecture is modular: the `api` module defines public interfaces (Table, Catalog, Scan, FileIO), `core` provides implementations and utilities, and separate modules handle engine-specific integrations and cloud backends. Key entry points include the Table interface for DDL/DML operations, Scan for query planning, FileIO for storage access, and the REST catalog client for remote metadata management. Data flows involve reading/writing manifest files, applying expressions for predicate pushdown, and coordinating commits via optimistic concurrency control. External dependencies include Apache Parquet, ORC, Avro, Arrow, and various cloud SDKs, but the core is designed to be lightweight and extensible.
The LLM identified 5 files worth highlighting for reuse:
A comprehensive expression framework for building, binding, evaluating, and optimizing filter predicates. Includes classes for literals, references, comparisons, logical operators, aggregates, and transforms. The Binder resolves unbound references against a schema, the Evaluator evaluates expressions against records, and the Inclusive/StrictMetricsEvaluators use column statistics for metadata-level filtering. The ExpressionVisitors enable custom traversal and rewriting. This module is self-contained and could be extracted for any data system needing a type-safe expression DSL.
Implements the Iceberg REST Catalog protocol, providing a remote catalog client that communicates with a RESTful server for table, namespace, and view operations. Includes HTTP client abstractions (HTTPRequest, HTTPResponse), retry strategies (ExponentialHttpRequestRetryStrategy), authentication managers (OAuth2, Basic, SigV4), and serialization for all request/response types. The RESTSessionCatalog manages sessions and caching. This module is a reusable reference implementation for any REST-based metadata service.
Provides file-level encryption using AES-GCM, with abstractions for key management (KmsClient), encrypted input/output files, and key metadata serialization. Includes standard encryption managers (StandardEncryptionManager) and native encryption support. The module is self-contained and demonstrates a clean separation of concerns between encryption algorithm, key management, and file I/O, making it reusable for any application needing transparent file encryption.
Integrates Apache Arrow with Iceberg's Parquet reader for vectorized columnar reads. Includes ArrowBatchReader, ArrowVectorAccessors, and VectorizedArrowReader that convert Parquet column data into Arrow vectors. The NullabilityHolder and ColumnVector manage nullability and vector storage. This module demonstrates non-obvious skill in bridging two columnar formats and is reusable for any project needing high-performance Arrow-based data access from Parquet files.
A generic bin packing algorithm implementation used for planning file rewrites (compaction). It packs items into bins based on a target size and weight function, with support for dynamic bin sizes and lookback. The algorithm is parameterized and not tied to Iceberg's domain, making it reusable for any resource allocation or packing problem. The code is clean, well-documented, and includes a test suite.