$ show cchenax/streamforge-ai
StreamForge AI is a real-time data platform designed for ML/AI workloads, integrating CDC ingestion, stream processing, feature storage, and serving. The architecture comprises distinct layers: ingestion (Debezium/Kafka), processing (Flink), storage (MinIO/S3 with optional Iceberg), and serving (online/offline feature store). Key components include a control plane API/UI for pipeline management, a Feast/Tecton-compatible feature store, schema registry client, drift detector, lineage tracker, and prefetch engine for ML training optimization. The platform supports end-to-end pipelines from database changes to feature materialization and inference, with examples demonstrating ML training, real-time serving, and feature consumption. While the system demonstrates strong modularity and integration of industry-standard tools, its operational complexity and polyglot nature (Java/Python/JS) present maintenance challenges.
The LLM identified 4 files worth highlighting for reuse:
Unified client for Confluent and Apicurio Schema Registries with automatic schema caching, retry logic, and Avro serialization/deserialization. Demonstrates robust integration patterns with fallback mechanisms and can be extracted as a standalone library for any project requiring schema registry integration.
Core feature store abstraction providing a unified interface for online (Redis) and offline (MinIO/S3, Iceberg) feature storage. Implements pluggable backend design, entity/feature versioning, and time-windowed aggregations. Can be decoupled from the platform and used independently in ML workflows requiring feature serving.
Implements Population Stability Index (PSI) for detecting drift in categorical and numerical features. Includes configurable windowing, baseline comparison, and alerting mechanisms. The detection logic is isolated and can be applied to any data stream monitoring scenario without dependencies on the broader platform.
Prefetch engine that analyzes historical access patterns to predict and preload hot data into cache before ML training execution. Uses frequency-based scoring and LRU eviction to optimize cache hit ratios. The algorithm is self-contained and adaptable to various caching systems beyond the demonstrated Redis implementation.