$ show add2ws/etl-engine-project
Etl-engine is a Java-based ETL (Extract, Transform, Load) library designed for high-performance data synchronization. It provides a lightweight, extensible framework built around three core components: Node (data sources/destinations/transformations), Pipe (thread-safe buffered queues for data transfer), and Dataflow (orchestrator for pipeline execution). The library supports various data sources including JDBC databases (PostgreSQL, MySQL, Oracle, ClickHouse, DuckDB) and file outputs, with built-in capabilities for batch processing, automatic error handling/retry, and detailed logging. The architecture follows a producer-consumer pattern using Java's concurrency utilities, enabling efficient parallel data processing. Sample implementations demonstrate common ETL scenarios like database-to-database transfers, conditional data routing, and value transformation.
The LLM identified 4 files worth highlighting for reuse:
Implements a thread-safe blocking queue for data transfer between ETL nodes using Java's ArrayBlockingQueue. Handles proper lifecycle management (start/stop) and provides a reusable pattern for any producer-consumer scenario. The component is self-contained with minimal dependencies (only java.util.concurrent) and solves a fundamental concurrency problem in data pipelines.
A lightweight data transfer object that holds column-value pairs in a map, with utility methods for JSON conversion and special row types (end/empty). This component solves the common problem of representing tabular data in ETL pipelines with minimal overhead. It's highly reusable due to its simplicity, lack of framework dependencies, and focused responsibility.
Implements proper CSV escaping according to RFC 4180 standards, correctly handling special characters like commas, quotes, and newlines. This utility addresses a common pain point in data export where naive CSV generation fails with embedded delimiters. The component is completely self-contained with no external dependencies and demonstrates attention to detail in data formatting.
Provides null-safe string comparison and manipulation utilities, including special handling for BigDecimal and Date objects to ensure accurate comparisons. This component solves common string comparison issues that arise in data processing (like trailing zeros in decimals or timezone differences in dates). It's reusable due to its focused scope, lack of dependencies, and solution to a ubiquitous problem in data engineering.