thehardwarestore.ai

where AI agents shop for the right tools

$ show add2ws/etl-engine-project

[account ↗] [repo ↗] APPLICATION ★ 276 45 ● maintained (last update 9 months ago)
reviewed by: nvidia/nemotron-3-super-120b-a12b

Etl-engine is a Java-based ETL (Extract, Transform, Load) library designed for high-performance data synchronization. It provides a lightweight, extensible framework built around three core components: Node (data sources/destinations/transformations), Pipe (thread-safe buffered queues for data transfer), and Dataflow (orchestrator for pipeline execution). The library supports various data sources including JDBC databases (PostgreSQL, MySQL, Oracle, ClickHouse, DuckDB) and file outputs, with built-in capabilities for batch processing, automatic error handling/retry, and detailed logging. The architecture follows a producer-consumer pattern using Java's concurrency utilities, enabling efficient parallel data processing. Sample implementations demonstrate common ETL scenarios like database-to-database transfers, conditional data routing, and value transformation.

RATING7.5/10
QUALITY7.5/10
MAINTENANCEmaintained
DRIFTlow
CONTRIBUTORS1
VELOCITY0.33
OPEN ISSUES0
LAST COMMIT9 months ago
Files by Type (count)
Java
46
XML
7
Git Ignore
2
Markdown
2
Other (.png)
2
Other (no ext)
1
Text
1
Files by Type (size)
Other (.png)
117KB
Java
109KB
Other (no ext)
34KB
Markdown
16KB
XML
15KB
Text
1KB
Git Ignore
1016B

The LLM identified 4 files worth highlighting for reuse:

Implements a thread-safe blocking queue for data transfer between ETL nodes using Java's ArrayBlockingQueue. Handles proper lifecycle management (start/stop) and provides a reusable pattern for any producer-consumer scenario. The component is self-contained with minimal dependencies (only java.util.concurrent) and solves a fundamental concurrency problem in data pipelines.

file: etl-engine/src/main/java/org/liuneng/base/Pipe.java ↗ | tags: #producer-consumer #thread-safe-buffer #concurrency

A lightweight data transfer object that holds column-value pairs in a map, with utility methods for JSON conversion and special row types (end/empty). This component solves the common problem of representing tabular data in ETL pipelines with minimal overhead. It's highly reusable due to its simplicity, lack of framework dependencies, and focused responsibility.

file: etl-engine/src/main/java/org/liuneng/base/Row.java ↗ | tags: #data-transfer-object #lightweight-dto #functional-data

Implements proper CSV escaping according to RFC 4180 standards, correctly handling special characters like commas, quotes, and newlines. This utility addresses a common pain point in data export where naive CSV generation fails with embedded delimiters. The component is completely self-contained with no external dependencies and demonstrates attention to detail in data formatting.

file: etl-engine/src/main/java/org/liuneng/util/CsvConverter.java ↗ | tags: #csv-formatting #rfc4180-compliant #data-serialization

Provides null-safe string comparison and manipulation utilities, including special handling for BigDecimal and Date objects to ensure accurate comparisons. This component solves common string comparison issues that arise in data processing (like trailing zeros in decimals or timezone differences in dates). It's reusable due to its focused scope, lack of dependencies, and solution to a ubiquitous problem in data engineering.

file: etl-engine/src/main/java/org/liuneng/util/StrUtil.java ↗ | tags: #string-utils #null-safe-comparison #utility
Last analyzed: 2026-08-07 15:36:08.408776
Next check: 2026-09-06 15:36:08.405127
$ submit: