$ show aws-samples/data-engineering-for-aws-immersion-day
This repository contains the lab materials for the AWS Data Engineering Immersion Day, a guided workshop demonstrating how to build a data lake and analytics pipeline on AWS using services like AWS Database Migration Service (DMS), AWS Glue, Amazon Athena, Amazon QuickSight, Amazon Redshift Spectrum, and AWS Lake Formation. The repo includes step-by-step PDF lab guides, CloudFormation templates for infrastructure provisioning, sample data loading scripts, and a self-contained anomaly detection example using Apache Flink with Random Cut Forest. While the Flink streaming job is a real application, the primary purpose of the repo is instructional, not a production deployment or a self-contained research project. The data lineage involves ingesting ticketing data from a PostgreSQL database via DMS into S3, transforming with Glue into Parquet, and analyzing with Athena/Redshift. Clickstream data is generated into Kinesis streams and processed in real-time with Kinesis Analytics (Flink) to detect anomalies. The repo is hybrid: it has both application code (Flink Java) and data pipeline orchestration, but the dominant character is a workshop with infrastructure-as-code and documentation.
The LLM identified 3 files worth highlighting for reuse:
A generic Flink `ProcessFunction` that integrates AWS Random Cut Forest for anomaly detection. It implements `CheckpointedFunction` to save and restore the RCF model state, uses a builder pattern for configuration, and accepts generic input/output type via `InputDataMapper` and `ResultMapper` functional interfaces. This component is isolated (only depends on Flink and RCF libraries), highly reusable for any streaming anomaly detection use case where a stateful ML model is applied per record.
A builder for `RandomCutForestOperator` that enforces validation on parameters (positive dimensions, sample size, etc.) and provides sensible defaults. Demonstrates a clean builder pattern with optional fields and required mappers. Can be extracted as a standalone pattern for constructing parameterized Flink operators.
Main entry point for a Flink streaming job that reads from Kinesis, applies Random Cut Forest anomaly detection, and sinks results back to Kinesis. Uses `KinesisAnalyticsRuntime` to read properties, has factory methods for source and sink, and demonstrates the recommended pattern for building AWS Kinesis Analytics applications with Flink. The code is clean and could serve as a template for other streaming jobs.