$ show spotify/dbeam
DBeam is a mature Java library and CLI tool that extracts data from SQL databases (PostgreSQL, MySQL, MariaDB, H2) and exports it as Apache Avro files, typically stored in Google Cloud Storage. It operates as a single-threaded Apache Beam pipeline, first inferring the table schema by reading one row, then streaming the full result set via JDBC directly into Avro binary encoding using a custom Beam sink. The tool supports partitioned exports, parallel query execution via a split column (experimental), multiple Avro codecs, KMS-decrypted passwords, and various array handling modes. It is designed for reliability in production batch data extraction pipelines and has been in use at Spotify since 2017. The codebase is well-tested and uses modern Java practices (AutoValue, Beam SDK 2.76), with CI/CD workflows for Maven Central releases.
The LLM identified 3 files worth highlighting for reuse:
Provides a builder-patterned decrypter for Google Cloud KMS encrypted passwords. Standalone class that configures KMS location, keyring, key, project, and optional HTTP transport/credentials. Decrypts base64-encoded ciphertext. Clean separation of concerns, reusable for any Java application needing KMS decryption with flexible credential scoping.
Maps JDBC SQL column types to Avro schema types, supporting logical types (timestamps, UUID), multiple array handling modes (bytes, typed from first row, Postgres metadata), and nullable union schemas. Handles edge cases like BIT precision, UUID with logicalType, and falls back to string for unknown types. The core logic is cleanly isolated and could be reused as a library for generating Avro schemas from JDBC ResultSet metadata.
Contains the algorithm for splitting a numeric range (min, max) into approximately equal subranges and generating SQL queries for parallel execution. The `generateRanges` method handles rounding up to avoid zero-size buckets, ensures inclusivity/exclusivity of endpoints, and respects a maximum parallelism count. Reusable for any project needing to partition work over a monotonically increasing column.