$ show DataTalksClub/data-engineering-zoomcamp
This is the DataTalksClub Data Engineering Zoomcamp — a free, instructor-led 9-week educational course repository, not a deployable application. It is a hybrid collection of teaching materials: Jupyter notebooks, dlt pipeline scripts, dbt models with tests, Airflow/Kestra/Spark/streaming examples, and workshop content. The dominant character is data_research: the repository exists to teach end-to-end data engineering through hands-on pipelines over NYC taxi trip data (yellow/green/FHV in CSV/Parquet from GCS) and the Open Library API. Data flows from raw sources through dlt extraction/normalization/loading into DuckDB/BigQuery, then through dbt transformations (staging/core layers) with singular/generic/unit tests and model contracts, plus batch processing with Spark and streaming with Kafka. The course is actively maintained with cohorts from 2022 through 2027, and includes a systematic image-distrust audit that removed 64 AI-generated illustrations from lessons.
The LLM identified 4 files worth highlighting for reuse:
A self-contained dlt pipeline that ingests the Open Library API via rest_api_source, demonstrating the three-stage extract-normalize-load pattern, incremental/append/merge write dispositions, and parameterized pipeline construction. Reusable as a template for any REST API ingestion into DuckDB or similar destinations with minimal adaptation.
A comprehensive dbt testing suite covering singular tests, custom generic tests, unit tests, source freshness checks, and model contracts — all expressed in YAML/SQL. These patterns are business-agnostic and can be extracted to enforce data-quality gates in any dbt project.
Parameterized Kestra flow definitions with conditional branches for taxi type and staging, deduplication via unique_row_id, and merge-based materialization. The YAML structure is a reusable orchestration pattern for scheduled, idempotent batch pipelines.
A Bruin-based pipeline definition with environment/connection configuration and asset-level materialization strategies. Demonstrates a modern, code-first alternative to Airflow for defining data pipelines, reusable as a scaffold for new data projects.