$ show cptn-io/el-cptn
This repository implements a self-hostable data integration platform for building, deploying, and managing ETL/ELT pipelines. The system features a polyglot microservices architecture: a React frontend (app-ux) for workflow visualization and configuration, two Spring Boot backend services (mgmt-svc for management APIs and ingestion-svc for event intake), and a Node.js processor for executing pipeline transformations. Core functionality includes defining sources, transformations, and destinations; scheduling pipelines via cron; monitoring execution metrics; role-based access control with SSO support; and a plugin architecture for community-contributed apps. Data flows involve ingestion services writing events to PostgreSQL queues, the processor executing transformation steps in sandboxed VMs, and delivering results to destinations. The platform emphasizes secure handling of sensitive data via encryption utilities and provides extensibility through dynamic script execution.
The LLM identified 5 files worth highlighting for reuse:
Isolated encryption utility using PBKDF2 key derivation and AES-GCM for secure storage of sensitive configuration values. Demonstrates proper cryptographic practices (IV handling, authentication tags) and has zero external dependencies beyond Java standard library, making it trivially extractable for any application needing encrypted secrets management.
Reusable filter parser that converts API query parameters (filtering, sorting, pagination) into structured predicates for database queries. Handles logical operators, field mapping, and type conversion in a framework-agnostic way. Could be dropped into any REST backend with minimal adaptation to replace ad-hoc query parsing logic.
Node.js module that safely executes user-provided JavaScript code (for pipeline transformations) using the vm2 sandbox. Properly restricts access to only allowed modules and globals, preventing escape vulnerabilities. This isolated component could be reused in any application requiring secure execution of untrusted scripts with minimal configuration changes.
Contains ConfigBuilder and HeaderBuilder React components for dynamically building configuration UIs (key-value pairs, nested objects) and HTTP header builders. Decoupled from application state, depends only on React and minor utilities. Demonstrates skill in handling complex nested structures and arrays in a user-friendly way, suitable for extraction into a standalone UI library.
Custom React hook combined with Context API for centralized toast notification management. Provides add/remove notification functions with deduplication and lifecycle handling. This notification system is completely decoupled from business logic and could be imported into any React project with minor prop adjustments.