$ show delta-io/delta
Delta Lake is a transactional table format and multi-engine connector suite. In production it writes Parquet data files alongside a JSON transaction log (`_delta_log`) and periodic checkpoints; every commit is an atomic append of log actions. Readers reconstruct table snapshots by replaying the log and checkpoints, enabling ACID operations, time travel, schema evolution, and incremental streaming on object storage. This repository is the reference implementation of that protocol, not an end-user data platform. The architecture is a multi-module Scala/Java/Python monorepo: `spark/` contains the Spark V1 and V2 connectors plus SQL extensions; `kernel/` is a standalone Java, engine-agnostic library exposing `Engine`, `TableManager`, `TransactionBuilder`, and a default Hadoop engine; `storage/` provides LogStore abstractions and cloud-specific commit coordination; and `flink/`, `iceberg/`, `sharing/`, and `python/delta` provide additional engine integrations. Key entry points are `DeltaTable`, `DeltaSparkSessionExtension`, `io.delta.kernel.TableManager`, and Flink's `DeltaDynamicTableSource`/`DeltaDynamicTableSink`. The central data flow is client -> `OptimisticTransaction`/Kernel `Transaction` -> log action -> LogStore commit; reads use `Snapshot`/`SnapshotBuilder` to replay log actions. External dependencies include Apache Spark, Hadoop FileSystem, Parquet, Jackson, and optional cloud SDKs. Maintenance is genuinely active: recent commits, CI matrices for Spark 4.0-4.2, Flink, Iceberg, Kernel, and Unity Catalog, plus a large compatibility test suite. The codebase nevertheless carries transitional complexity: a V2/Kernel-backed Spark connector coexists with the legacy V1 path, many Spark-version shims exist, test suites are partly auto-generated, and the old `connectors/` directory is explicitly in maintenance mode. Those factors prevent a perfect score, but this is a mature, high-craft storage-format project.
The LLM identified 4 files worth highlighting for reuse:
Defines the connector contract for Delta Kernel: JSON handling, Parquet handling, file-system access, and expression evaluation. Any engine (Hadoop-based, native, or in-memory) can implement this interface, making it the clean seam for building new Delta readers and writers without depending on Spark or Flink internals.
Wraps iterators with closeable semantics and combinators such as `map`, `flatMap`, and `withClose`. It prevents file and stream leaks in long-running scans and can be lifted into any Java/Scala data-processing pipeline where lazily-evaluated resources need deterministic cleanup.
Represents Kernel scan state as an immutable `Row` with a fixed schema and ordinal-based typed accessors. This is a reusable pattern for carrying complex configuration or internal state as schema'd data, and it decouples state representation from engine-specific classes.
An iterator that detects and exposes contiguous version ranges from a versioned log, which is important for CDC and streaming offset tracking. It is a small, self-contained utility with minimal internal coupling, making it reusable for any log-replay or sequence-validation task.