Haystack is an open-source Python framework for building production-ready LLM applications. It provides a modular, component-based architecture where data processing steps (converters, embedders, retrievers, generators, evaluators, etc.) are connected into pipelines or agents. The framework includes over 100 built-in components, support for 50+ integrations (OpenAI, Anthropic, Azure, HuggingFace, etc.), async execution, streaming, serialization, evaluation, and observability. It is designed for RAG, semantic search, question answering, autonomous agents, and multimodal applications. The code is well-structured with consistent patterns (component decorator, lifecycle methods, serialization protocols), extensive test coverage, and security hardening (fuzzing, sandboxed Jinja2, deserialization allowlists).
OpenMontage is an open-source, agentic video production platform where an AI coding assistant (Claude Code, Cursor) orchestrates the entire pipeline. It provides a library of 100+ tools (video generation, audio, graphics, enhancement, analysis, avatar, subtitles) behind a scored provider selection layer, and 10+ production pipeline manifests (animated explainer, cinematic, documentary montage, talking head, etc.) that define stage order, approval gates, and tool assignments. The core architecture is a Python framework with a BaseTool class, ToolRegistry for auto-discovery, pipeline loader, checkpoint system, and a Remotion-based video composer. It also includes a Backlot living storyboard server for live production monitoring and an Ink Theater procedural puppetry engine. While primarily an application, it contains data/research elements such as ManimGL/ManimCE educational animation examples covering transformer internals, quantum mechanics, and ML concepts, as well as a CLIP-based corpus builder for documentary montage pipelines.
Spinnaker is a continuous delivery platform with a microservices architecture for deploying and managing applications across multiple cloud providers. The system consists of distinct services including clouddriver (cloud abstraction), orca (pipeline orchestration), deck (React/UI), gate (API gateway), fiat (authorization), front50 (storage), echo (event processing), igor (CI integrations), kayenta (canary analysis), and rosco (image baking). It features polyglot implementation (Java/Kotlin/TypeScript/Groovy), Spring Boot backend services, React frontend, RESTful APIs, event-driven architecture, and extensive cloud provider integrations. The platform enables pipeline definition, execution, artifact management, and deployment strategies across AWS, Kubernetes, GCP, Azure, and other environments.
GlassFlow ClickHouse ETL is a full-stack, open-source streaming ETL platform designed to ingest data from Apache Kafka (or OpenTelemetry), apply in-flight transformations (deduplication, temporal joins, filtering, stateless transformations), and sink the processed data into ClickHouse with schema evolution support. The system provides a REST API for pipeline lifecycle management, a web UI for visual configuration, and a NATS JetStream backbone for reliable message passing. It supports advanced features like dead-letter queues, schema registry integration (Avro/Protobuf), OpenTelemetry receiver (gRPC/HTTP), and Kubernetes-native deployment via Helm charts. The architecture follows a microservice pattern with distinct roles: ingestor, dedup, join, sink, and OTLP receiver, each running as an independent binary. The Go backend uses a clean component pattern with pluggable processors (filter, dedup, transform) in a chain, backed by PostgreSQL for configuration persistence and BadgerDB for deduplication state. The React/Next.js frontend features a multi-step wizard for pipeline creation, real-time status updates via SSE, and extensive notification channels (toast, modal, banner, Slack, email). The codebase demonstrates high engineering rigor with comprehensive testing (Go unit/integration/BDD tests, Vitest for UI), observability instrumentation (OpenTelemetry, Prometheus metrics), and a design-system approach with Figma tokens. While primarily an application, the repo includes data-focused demos (5G telemetry with OTLP emitter, fraud detection) and a suite of ClickHouse table creation scripts and verification queries, indicating a hybrid character with strong data engineering concerns. The platform specifically targets the challenges of CDC ingestion, backfill, schema evolution, and data quality for ClickHouse sinks.
Elyra is a JupyterLab extension suite that provides a visual pipeline editor for building, validating, and executing multi-step AI/ML workflows on Kubeflow Pipelines, Apache Airflow, or locally. It includes enhanced script editors (Python, R, Scala) with kernel management and debugging, a code snippet manager, and a metadata-driven system for runtime configurations, runtime images, and component catalogs. The backend (Python) handles pipeline parsing, validation, code generation via Jinja2 templates, and submission to remote orchestrators, while the frontend (TypeScript/React) delivers a rich UI with drag-and-drop, property editing, and real-time validation. The architecture is modular with pluggable runtime processors, component catalog connectors, and metadata storage backends, making it extensible for custom runtimes and data sources. Although primarily an application, it serves the MLOps domain by orchestrating data science notebooks and scripts as reproducible pipelines.
Certd is a full-stack web application that automates the entire lifecycle of SSL/TLS certificates, from application via the ACME protocol to deployment across 110+ cloud providers, CDNs, and hosting platforms. It utilizes a pipeline engine for orchestrating multi-stage tasks (cert apply, conversion, deploy, notification) with retry, skip-on-success, and cron/webhook triggers. The system features a plugin architecture with decorator-based registration for tasks, access credentials, DNS providers, notifications, and OAuth, making it highly extensible. The monorepo includes a Vue 3 + Vben admin frontend and a Midway.js (Koa) backend with TypeORM and multi-database support (SQLite, MySQL, PostgreSQL). It also supports commercial features like subscription plans, wallet, payments, and enterprise project isolation. While the codebase is well-structured and actively maintained, the large number of cloud provider plugins and legacy auto-generated SDKs (e.g., JDCloud) introduce maintenance overhead and potential API drift.
This repository is an AI-driven story-to-video production platform. It is a full-stack Django 3.2 backend with Vue 2 frontend, Celery task queue, Redis pub/sub, and Docker deployment. The system ingests user story ideas and orchestrates a multi-stage pipeline: rewrite → storyboard → image generation (text2image/image2video) → camera movement → video generation → Jianying (剪映) draft export. All AI capabilities are consumed via external APIs (OpenAI, ComfyUI, Volcano Engine, etc.) through a pluggable AI client abstraction layer. Key entry points include RESTful APIs (DRF), SSE streaming for real-time progress, MCP (Model Context Protocol) JSON-RPC endpoints for agent integration, and a Vue-based canvas UI for visual workflow editing. The architecture follows a modular Django app pattern with clear separation: core infrastructure (pipeline orchestration, AI clients, Redis streaming), domain apps (projects, prompts, models, workflows, content, agent, MCP), and a frontend SPA. External dependencies include Redis, Celery, MySQL/PostgreSQL, external AI vendor APIs, and optional Jianying draft generation. The codebase demonstrates strong engineering practices with reusable patterns for AI client abstraction, Redis pub/sub streaming, MCP tool registration, and pipeline stage processors, but has notable technical debt: Django 3.2 (EOL), mixed async/gevent, Vue 2 (EOL), and some closed-source compiled components.
This repository implements a self-hostable data integration platform for building, deploying, and managing ETL/ELT pipelines. The system features a polyglot microservices architecture: a React frontend (app-ux) for workflow visualization and configuration, two Spring Boot backend services (mgmt-svc for management APIs and ingestion-svc for event intake), and a Node.js processor for executing pipeline transformations. Core functionality includes defining sources, transformations, and destinations; scheduling pipelines via cron; monitoring execution metrics; role-based access control with SSO support; and a plugin architecture for community-contributed apps. Data flows involve ingestion services writing events to PostgreSQL queues, the processor executing transformation steps in sandboxed VMs, and delivering results to destinations. The platform emphasizes secure handling of sensitive data via encryption utilities and provides extensibility through dynamic script execution.