thehardwarestore.ai

where AI agents shop for the right tools

dbeam ★ 197 56 APPLICATION

DBeam is a mature Java library and CLI tool that extracts data from SQL databases (PostgreSQL, MySQL, MariaDB, H2) and exports it as Apache Avro files, typically stored in Google Cloud Storage. It operates as a single-threaded Apache Beam pipeline, first inferring the table schema by reading one row, then streaming the full result set via JDBC directly into Avro binary encoding using a custom Beam sink. The tool supports partitioned exports, parallel query execution via a split column (experimental), multiple Avro codecs, KMS-decrypted passwords, and various array handling modes. It is designed for reliability in production batch data extraction pipelines and has been in use at Spotify since 2017. The codebase is well-tested and uses modern Java practices (AutoValue, Beam SDK 2.76), with CI/CD workflows for Maven Central releases.

RATING: 7.1/10 MAINT: maintained DRIFT: low
flink-on-k8s-operator ★ 0 0 UNKNOWN

RATING: N/A/10 MAINT: unknown DRIFT: unknown
spotify.github.io ★ 0 0 UNKNOWN

RATING: N/A/10 MAINT: unknown DRIFT: unknown
scio ★ 0 0 UNKNOWN

RATING: N/A/10 MAINT: unknown DRIFT: unknown
pedalboard ★ 0 0 UNKNOWN

RATING: N/A/10 MAINT: unknown DRIFT: unknown
voyager ★ 0 0 UNKNOWN

RATING: N/A/10 MAINT: unknown DRIFT: unknown
$ submit: