$ show glassflow/clickhouse-etl
GlassFlow ClickHouse ETL is a full-stack, open-source streaming ETL platform designed to ingest data from Apache Kafka (or OpenTelemetry), apply in-flight transformations (deduplication, temporal joins, filtering, stateless transformations), and sink the processed data into ClickHouse with schema evolution support. The system provides a REST API for pipeline lifecycle management, a web UI for visual configuration, and a NATS JetStream backbone for reliable message passing. It supports advanced features like dead-letter queues, schema registry integration (Avro/Protobuf), OpenTelemetry receiver (gRPC/HTTP), and Kubernetes-native deployment via Helm charts. The architecture follows a microservice pattern with distinct roles: ingestor, dedup, join, sink, and OTLP receiver, each running as an independent binary. The Go backend uses a clean component pattern with pluggable processors (filter, dedup, transform) in a chain, backed by PostgreSQL for configuration persistence and BadgerDB for deduplication state. The React/Next.js frontend features a multi-step wizard for pipeline creation, real-time status updates via SSE, and extensive notification channels (toast, modal, banner, Slack, email). The codebase demonstrates high engineering rigor with comprehensive testing (Go unit/integration/BDD tests, Vitest for UI), observability instrumentation (OpenTelemetry, Prometheus metrics), and a design-system approach with Figma tokens. While primarily an application, the repo includes data-focused demos (5G telemetry with OTLP emitter, fraud detection) and a suite of ClickHouse table creation scripts and verification queries, indicating a hybrid character with strong data engineering concerns. The platform specifically targets the challenges of CDC ingestion, backfill, schema evolution, and data quality for ClickHouse sinks.
The LLM identified 3 files worth highlighting for reuse:
A reusable Go component that provides batch publishing to NATS JetStream with context-based cancellation, configurable batch size and timeout, and a retry/backoff mechanism. It abstracts away JetStream connection management, message batching logic, and error handling, making it easy to integrate NATS publish-async into any streaming pipeline. The implementation demonstrates clean separation of concerns and is well-tested alongside its reader counterpart.
Flattens OpenTelemetry metrics (gauge, sum, histogram, summary) into a uniform internal message format. Handles the complexity of OTLP protobuf structures (data points, attributes, exemplars) and converts them into a simple JSON-like representation for downstream processing. The module is self-contained, requires only the models package, and can be extracted for any project needing to normalize OTLP metrics into a schema-agnostic form.
A transformation engine that compiles and caches expr-lang programs keyed by source schema version, avoiding recompilation overhead for repeated transformations on the same schema. It falls back to a pass-through transform when no transformation is configured. This pattern of versioned, cached compilation is applicable to any data pipeline requiring schema-aware, deterministic transformations with performance guarantees.