Apache Airflow is a workflow orchestration platform for authoring, scheduling, and monitoring data pipelines. The system uses a modular architecture with core components, extensible providers, a React/TypeScript web UI, and a FastAPI-based REST API. Key entry points include the CLI (`airflow` command), web server, scheduler, and triggerer. Data flows involve DAG parsing, task scheduling via executors, and state persistence in a metadata database. External dependencies include PostgreSQL/MySQL, Redis/RabbitMQ for Celery, and various provider-specific integrations. The codebase demonstrates strong engineering practices with version compatibility patterns, plugin systems, and well-structured API layers.
This is an Astronomer-maintained Helm chart for deploying Apache Airflow on Kubernetes with custom extensions like git-sync relay, dag-deploy server, auth/logging sidecars, and pgbouncer. It wraps the upstream Apache Airflow chart as a subchart and adds robust security hardening (Pod Security Standards restricted), ingress support, and OpenShift compatibility. The repository includes an extensive test suite that validates rendered templates against Kubernetes API schemas across multiple versions (1.31-1.35) and runs functional tests on kind clusters. The chart is well-structured with modular Helm templates, helper functions for common patterns (private CA trust, sidecar injection, service account naming), and comprehensive CI/CD pipelines. It is actively maintained and used by Astronomer to distribute Airflow via their Helm repository.
This repository contains Helm charts and tooling for deploying the Astronomer Platform (a Kubernetes-hosted enterprise data platform for Apache Airflow) into Kubernetes clusters. It provides a modular architecture with subcharts for core components (Houston API, Commander, Navigator, Pilot, Registry) and infrastructure (NGINX, Prometheus, Grafana, Alertmanager, Elasticsearch, Vector, PostgreSQL, NATS). The repository supports multiple deployment topologies (control plane, data plane, unified) and includes enterprise features like high availability, network policies, RBAC, and external secret management. Extensive testing validates chart rendering and functional behavior across Kubernetes versions, with automation scripts for local development, version migration, and cluster setup.
Elyra is a JupyterLab extension suite that provides a visual pipeline editor for building, validating, and executing multi-step AI/ML workflows on Kubeflow Pipelines, Apache Airflow, or locally. It includes enhanced script editors (Python, R, Scala) with kernel management and debugging, a code snippet manager, and a metadata-driven system for runtime configurations, runtime images, and component catalogs. The backend (Python) handles pipeline parsing, validation, code generation via Jinja2 templates, and submission to remote orchestrators, while the frontend (TypeScript/React) delivers a rich UI with drag-and-drop, property editing, and real-time validation. The architecture is modular with pluggable runtime processors, component catalog connectors, and metadata storage backends, making it extensible for custom runtimes and data sources. Although primarily an application, it serves the MLOps domain by orchestrating data science notebooks and scripts as reproducible pipelines.
Ro-DOU is an Apache Airflow-based system for automated clipping and monitoring of Brazilian official gazettes, primarily the Diário Oficial da União (DOU) and municipal gazettes via Querido Diário. It dynamically generates Airflow DAGs from declarative YAML configuration files, allowing users to define search terms, notification channels (email, Slack, Discord, Apprise), and optional AI-powered summaries using OpenAI, Gemini, Claude, or Azure. The architecture is modular: a central DAG generator creates workflows that execute search tasks via multiple backends (DOU API, INLABS OpenSearch/SQL, Querido Diário API, DOESP API), group results, and deliver formatted reports. The system includes a CLI and web-based YAML generator for configuration, extensive documentation, and Helm charts for Kubernetes deployment. While robust in functionality, it relies on several external APIs with fragile parsing logic and maintains legacy code paths (e.g., INLABS SQL mode), introducing moderate technical drift. Reusable components include a clean notification interface pattern, an OpenSearch query builder, and a provider-agnostic AI runner.