$ show DataExpert-io/data-engineer-handbook
This repository is a comprehensive educational resource for data engineering, structured as a handbook and bootcamp. It contains curated learning paths (beginner, intermediate, Databricks AI bootcamp) with lecture materials, homework, and sample datasets. The codebase includes teaching examples of core data engineering patterns: dimensional data modeling (cumulative tables, SCD type 2), fact data modeling (array metrics, datelist int), Apache Spark jobs with unit tests, Apache Flink streaming jobs, and analytical SQL patterns (retention, funnel, grouping sets, growth accounting). It also includes a small Flask server for A/B experimentation and Docker Compose setups for local development. While primarily educational, the SQL scripts and job templates are production-quality and reusable across projects. The repository is actively maintained with pinned dependencies and clear documentation, making it highly reproducible for learners.
The LLM identified 5 files worth highlighting for reuse:
Implements a Slowly Changing Dimension (SCD) Type 2 backfill and incremental generation query using window functions and array aggregation. Demonstrates a reusable pattern for tracking historical changes in dimension tables, with idempotent logic that can be adapted to any SCD scenario.
Generates monthly array metrics by aggregating daily event data into a cumulative array of metric values per user. Uses PostgreSQL array functions to create a compact, queryable representation of user behavior over time. This pattern is reusable for any time-series aggregation where array-based storage is preferred over wide tables.
A PySpark ETL job that implements SCD Type 2 logic using Spark SQL and DataFrame API. The job is structured as a reusable template: a transformation function that takes a SparkSession and parameters, a main entry point, and a test suite using chispa for DataFrame comparison. Demonstrates clean separation of concerns and testability for Spark jobs.
A Flink streaming job that reads from Kafka, applies a tumbling window aggregation, and writes results to PostgreSQL. Uses Flink's Table API and SQL for declarative processing. The job is parameterized via environment variables and follows a clean structure (source, transformation, sink). Reusable as a template for any windowed aggregation pipeline.
Performs user retention analysis using SQL window functions and self-joins to compute weekly retention cohorts. The query is optimized with proper indexing hints and demonstrates a standard pattern for calculating retention curves from event logs. Easily adaptable to any event-based retention analysis.