Apache Iceberg is a high-performance table format for analytic data lakes, providing ACID transactions, snapshot isolation, schema evolution, and time travel. The repository contains the Java reference implementation, including core libraries for table metadata management, file format support (Parquet, ORC, Avro, Arrow), expression evaluation, encryption, and a REST catalog protocol. It integrates deeply with Spark (versions 3.5, 4.0, 4.1, 4.2), Flink (1.20, 2.2, 2.3), Kafka Connect, Hive, and cloud object stores (AWS S3, Azure ADLS, Aliyun OSS, GCS). The architecture is modular: the `api` module defines public interfaces (Table, Catalog, Scan, FileIO), `core` provides implementations and utilities, and separate modules handle engine-specific integrations and cloud backends. Key entry points include the Table interface for DDL/DML operations, Scan for query planning, FileIO for storage access, and the REST catalog client for remote metadata management. Data flows involve reading/writing manifest files, applying expressions for predicate pushdown, and coordinating commits via optimistic concurrency control. External dependencies include Apache Parquet, ORC, Avro, Arrow, and various cloud SDKs, but the core is designed to be lightweight and extensible.
This repository is Confluent's official JavaScript client for Apache Kafka, wrapping the high-performance C library librdkafka via native Node.js addons. It provides two API surfaces: a legacy callback-based API (node-rdkafka) and a modern promisified API compatible with the popular KafkaJS library. The client supports all core Kafka operations: producing and consuming messages (including transactional and exactly-once semantics), admin operations (topic management, consumer groups, offsets), and stream-based interfaces. Additionally, it includes a full Schema Registry client (TypeScript) with serializers/deserializers for Avro, JSON Schema, and Protobuf, along with a rule execution engine supporting CEL (Common Expression Language), JSONata, and field-level encryption via pluggable KMS drivers (AWS, GCP, Azure, HashiCorp Vault, local). The architecture is layered: a C++ binding layer handles the Kafka protocol, while pure JavaScript/TypeScript implements schema management, rule evaluation, and encryption. Key entry points are `lib/index.js` (node-rdkafka), `lib/kafkajs/index.js` (KafkaJS), and `schemaregistry/index.ts` (Schema Registry). External dependencies include librdkafka, protobuf, avsc, axios, and cloud KMS SDKs. The codebase is production-grade with extensive unit, integration, and e2e tests, performance benchmarks, and CI/CD pipelines across multiple platforms.
DBeam is a mature Java library and CLI tool that extracts data from SQL databases (PostgreSQL, MySQL, MariaDB, H2) and exports it as Apache Avro files, typically stored in Google Cloud Storage. It operates as a single-threaded Apache Beam pipeline, first inferring the table schema by reading one row, then streaming the full result set via JDBC directly into Avro binary encoding using a custom Beam sink. The tool supports partitioned exports, parallel query execution via a split column (experimental), multiple Avro codecs, KMS-decrypted passwords, and various array handling modes. It is designed for reliability in production batch data extraction pipelines and has been in use at Spotify since 2017. The codebase is well-tested and uses modern Java practices (AutoValue, Beam SDK 2.76), with CI/CD workflows for Maven Central releases.