SQLFluff is a mature, actively maintained SQL linter and auto-formatter with production scope far beyond a simple style checker. It exposes both a Click-based CLI and a Python API (lint, fix, parse, list_rules, list_dialects) and supports over 30 SQL dialects, including ANSI, BigQuery, Snowflake, Postgres, T-SQL, Oracle, Databricks, ClickHouse, and DuckDB. The core pipeline discovers files, optionally expands Jinja/dbt/SQLMesh templating, lexes and parses SQL into a segment tree, then applies configurable rule crawlers that emit lint violations and in many cases generate auto-fixes. The implementation is layered: a public API and CLI wrap a core linter, a grammar-combinator parser, a rule framework with fix primitives, a hierarchical config system, and a pluggy-based plugin host. A Rust workspace (sqlfluffrs) provides an optional high-performance lexer/parser and partial Rust-native rule detection, with extensive Python/Rust parity tests to keep behavior aligned.
OpenDBT is a Python library and CLI tool that extends dbt-core functionality through runtime patching. It provides enhanced capabilities including custom adapter integration, local Python model execution, DLT job integration, Airflow orchestration, multi-project (DBT Mesh) support, and an advanced documentation UI with column-level lineage. The architecture uses runtime patching to modify dbt's behavior without changing its source code, featuring a modular design with version-specific adaptations for dbt v1.7 and v1.8+, shared components for cross-version functionality, and dedicated integrations for Airflow and catalog generation. Entry points include the 'opendbt' CLI command, OpenDbtProject Python API, and Airflow operators. Data flows involve user CLI commands triggering enhanced dbt executions, documentation generation creating catalogl.json with lineage data, and Airflow operators orchestrating dbt tasks within DAGs.
This is the DataTalksClub Data Engineering Zoomcamp — a free, instructor-led 9-week educational course repository, not a deployable application. It is a hybrid collection of teaching materials: Jupyter notebooks, dlt pipeline scripts, dbt models with tests, Airflow/Kestra/Spark/streaming examples, and workshop content. The dominant character is data_research: the repository exists to teach end-to-end data engineering through hands-on pipelines over NYC taxi trip data (yellow/green/FHV in CSV/Parquet from GCS) and the Open Library API. Data flows from raw sources through dlt extraction/normalization/loading into DuckDB/BigQuery, then through dbt transformations (staging/core layers) with singular/generic/unit tests and model contracts, plus batch processing with Spark and streaming with Kafka. The course is actively maintained with cohorts from 2022 through 2027, and includes a systematic image-distrust audit that removed 64 AI-generated illustrations from lessons.
Mage is an open-source data pipeline orchestration platform that combines a visual notebook-style editor with a production-grade scheduling and execution engine. The system enables users to build, run, and monitor data pipelines composed of Python, SQL, R, and dbt blocks. Architecturally, it is a full-stack web application with a Python backend (Tornado-based server, SQLAlchemy ORM, REST API) and a Next.js/React frontend with a custom component library. Key entry points include the CLI (`mage` command), the web UI served by `mage_ai/server/server.py`, and the API layer at `mage_ai/api/` which exposes CRUD resources for pipelines, blocks, runs, schedules, users, and integrations. Data flows center around pipeline execution: blocks are connected in a DAG, each block can be a data loader, transformer, or exporter. Execution is handled by a configurable executor (local, K8s, EMR, PySpark) and scheduled via a built-in scheduler or external triggers. The platform integrates with dozens of data sources and destinations through a Singer-based integration framework, supports streaming pipelines, and includes advanced features like RBAC, version control, AI-assisted pipeline generation, and a command center for global search. It depends heavily on external services such as cloud storage (S3, GCS, Azure), databases (PostgreSQL, BigQuery, Snowflake, etc.), and authentication providers (OAuth, LDAP, OIDC).