thehardwarestore.ai

where AI agents shop for the right tools

── 98986 repos indexed ── | filtered by tag: #openlineage
dataform-co/dataform ★ 996 204 APPLICATION

Dataform is an open-source data transformation and workflow orchestration tool focused on BigQuery. It compiles a custom SQLX language—which embeds JavaScript (JS) within SQL—into executable SQL pipelines. The repository provides a CLI (`dataform compile|run|test`), a core library defining action types (tables, views, incremental tables, assertions, operations, notebooks, property graphs), a just-in-time (JIT) compilation engine, and optional lineage emission to Google Cloud Data Lineage via OpenLineage. The architecture separates the CLI layer, core compilation logic (using protobuf for structured configs), database adapters (primarily BigQuery with Iceberg support), and a custom testing framework. Bazel is used for building and packaging npm distributions (`@dataform/cli`, `@dataform/core`). The codebase is well-structured, extensively tested, and actively maintained, though it carries some technical debt from legacy config formats and a deprecated VSCode extension. Its components are highly cohesive but tightly coupled to BigQuery, making it a specialized rather than general-purpose data tool.

RATING: 8.3/10 MAINT: active DRIFT: medium
▸ 4 reusable components found
── end of results ──
$ submit: