Apache Airflow is a workflow orchestration platform for authoring, scheduling, and monitoring data pipelines. The system uses a modular architecture with core components, extensible providers, a React/TypeScript web UI, and a FastAPI-based REST API. Key entry points include the CLI (`airflow` command), web server, scheduler, and triggerer. Data flows involve DAG parsing, task scheduling via executors, and state persistence in a metadata database. External dependencies include PostgreSQL/MySQL, Redis/RabbitMQ for Celery, and various provider-specific integrations. The codebase demonstrates strong engineering practices with version compatibility patterns, plugin systems, and well-structured API layers.
RocketRide is a high-performance data processing engine with modular nodes, AI capabilities, and cross-platform clients. The system provides a visual interface for building data pipelines that integrate various data sources, AI/ML services, and processing nodes. At its core, it features a C++-based execution engine for performance, Python-based processing nodes for flexibility, and multiple client applications (web, desktop, VS Code extension) for interaction. The platform emphasizes extensibility through its node system, allowing users to create custom processing components while providing built-in integrations for databases, cloud storage, AI services, and popular APIs. It includes monitoring, observability, and deployment capabilities for production use.
This repository is a curated directory of internet radio streams, organized by genre, with automated tooling to generate M3U playlists, check link health, and probe stream availability. It is primarily a data curation project, with the README.md serving as the source of truth. The supporting scripts form a robust data pipeline: parsing the markdown, generating playlists, and running health checks with retry logic and silence detection. The project demonstrates strong data engineering practices in a non-traditional data context, with shared parsers, persistent state tracking, and CI workflows that handle race conditions gracefully.
TradingAgents is a multi-agent LLM framework for financial trading research and backtesting. It orchestrates specialized agents (analysts, researchers, debaters, trader, portfolio manager) through a LangGraph state machine to produce trading decisions. The system emphasizes point-in-time data isolation to prevent lookahead bias, supports multiple LLM providers and data vendors, and includes a comprehensive backtesting engine. It is structured as a Python library with a rich CLI interface. The architecture is a hybrid of a software application and a research framework. It demonstrates high craftsmanship in its data pipeline (vendor routing, symbol normalization, date windowing), LLM integration (capability registry, structured output fallback), and concurrency handling (atomic file writes). While the core abstractions are clean and reusable, the codebase is tightly coupled to the specific behaviors of its external dependencies (LLM APIs, Yahoo Finance, SEC EDGAR), making it sensitive to ecosystem drift. The extensive test suite provides good coverage for unit and integration scenarios, but the reliance on mocking means regressions in vendor adapters may go undetected without manual integration runs.
genblaze is a production-grade Python monorepo SDK for orchestrating generative AI media pipelines — image, video, audio, text-to-speech, and chat — across 16+ provider integrations (OpenAI, Google Gemini/Imagen/Veo, NVIDIA NIM, GMICloud, Runway, Luma, Replicate, FAL, ElevenLabs, AssemblyAI, Hume, LMNT, Stability Audio, Decart, AtlasCloud). The public API is a chainable Pipeline/Step abstraction with batch, streaming, ingest, and agent-loop modes, plus a Click-based CLI (extract/verify/replay/index) for manifest inspection, a schema-first spec package generating TypeScript types from JSON Schemas, and ~30 runnable examples. Architecturally it is a monorepo: libs/core (genblaze-core) contains the framework (pipeline, providers, model registry, storage, media handlers, canonical hashing, signing, observability, webhooks), while each connector under libs/connectors/* is an independently versioned package registering itself via plugin entry points. Key entry points are Pipeline.run()/batch_run()/stream() for programmatic use and the genblaze command line; data flows through provider submit/poll/fetch_output lifecycles, optional caching, S3/B2 object-storage sinks with URL policies, and manifest generation that can be embedded directly into output media files. Two capabilities set this apart. First, provenance: every run produces a canonical, hash-verified Manifest (SHA-256 over deterministically serialized JSON), optionally Ed25519-signed, and byte-preservingly embedded into JPEG/PNG/WebP/MP4/audio files via XMP/atom splicing without re-encoding. Second, security and reliability engineering: SSRF protection with DNS pinning, credential redaction, ReDoS-safe pattern heuristics, ffmpeg injection escaping, configurable retry policies, model liveness probes, and discovery-aware validation (NATIVE/PARTIAL/NONE support levels) that reduces hardcoded slug rot. The project is actively maintained with unusually thorough tooling: golden hash-stability tests, a provider conformance test base class (~15 contract checks per connector), pin-parity and PyPI-metadata release gates, zizmor/dependabot CI, and live probe scripts that fail on API drift. Main risks are the inherent churn of 16+ third-party APIs (mitigated but not eliminated by mock-heavy tests) and a package/versioning layout complex enough that installs rely on documented make targets. Overall this is a strong, well-architected application SDK — suitable both for direct use and as a reference for multi-provider integration frameworks.
WhoDB is a self-hosted, multi-database workspace and management platform by Clidey (Apache-2.0). The production system is a Go backend (chi router, gqlgen GraphQL API, BAML-generated LLM clients) served alongside a React/TypeScript SPA (Vite, Apollo, Redux), plus a Cobra/Bubbletea CLI/TUI, a Wails desktop app, an MCP server exposing database tools to AI agents, and multi-language SDKs (Python, TypeScript, Go, Rust, Java) targeting the hosted platform API. It lets users browse/edit/query rows, inspect schemas and relationships, run arbitrary SQL/NoSQL commands, import/export CSV and Excel, generate mock data, and chat with an AI assistant that produces SQL, explains queries, and executes verified operations. Supported engines include PostgreSQL, MySQL/MariaDB, SQLite, CockroachDB, TiDB, DuckDB, ClickHouse, QuestDB, YugabyteDB, MongoDB, Redis/Valkey/Dragonfly, Elasticsearch/OpenSearch, and Memcached, with automatic discovery of cloud-hosted instances on AWS, Azure, and GCP. Architecturally the backend is a database-plugin registry: every engine implements a PluginFunctions interface, mostly by embedding a shared GORM-based SQL plugin and overriding dialect specifics (constraint parsing, batch operations, SQL builders); non-relational stores use native adapters. Security is treated seriously: a SQL guard classifies commands as read-only vs mutating across SQL/MongoDB/Redis, egress validation blocks SSRF targets including cloud-metadata IPs, sessions use hashed cookie tokens with CSRF double-submit, and audit logging classifies events into telemetry/access/compliance. The frontend uses registry patterns (component-registry, route-registry, sidebar-registry) to keep CE/EE edition differences declarative. The most reusable pieces are security components (sqlguard read-only classification, egress SSRF filter), the mock-data generator's column-pattern matching, and the schema-diff engine — all self-contained enough to extract into other projects. The codebase is actively maintained with extensive CI/CD (Docker, npm, Snap, Windows Store, macOS releases), per-database e2e suites, and accessibility checks. Weaknesses are inherent complexity: five SDKs must stay synchronized with a generated manifest, BAML generation introduces platform-specific build stubs, cloud-provider connectivity logic is duplicated across AWS/GCP/Azure, and the broad plugin interface gives new database adapters a large implementation surface.
The repository is a data pipeline that generates Shadowrocket configuration files for selective traffic routing, particularly targeting users in Russia who need to bypass censorship while maintaining access to local services. It automates the collection, processing, and formatting of domain and IP lists from various sources into Shadowrocket-compatible rule sets and configuration files. The pipeline fetches data from diverse sources including GitHub repositories, websites (like the Central Bank of Russia), and manually curated lists. It then processes this data through a series of transformations: filtering, formatting, IDN conversion, section-based processing, and DNS resolution, to produce standardized output files that adhere to Shadowrocket's rule syntax. The processed data is used to generate configuration files that define which traffic should go through proxies and which should connect directly. Key reusable components within the pipeline include an IDN decoding script that properly handles internationalized domain names by detecting non-ASCII characters and converting to punycode while preserving original formatting, and a section-based list processor that applies different transformations to file segments based on markers. These components demonstrate non-obvious skill in handling specific data challenges and follow patterns that could be extracted for use in other data processing projects. The pipeline is executed on a daily schedule via GitHub Actions, ensuring the generated configuration files remain up-to-date with the latest domain and IP information from the source feeds. This automation allows users to benefit from continuously improved routing rules without manual intervention.
slacrawl is a local-first Slack archiving CLI tool written in Go. It ingests messages, files, users, and channels from multiple sources—Slack API (bot/user tokens), Slack Desktop local cache (LevelDB + IndexedDB), MCP (Model Context Protocol) adapters, and external provider subprocesses—and stores them in a local SQLite database with full-text search (FTS5). The tool provides a rich command surface including sync, search, TUI browsing, analytics reports (digest, trends, quiet channels), retention purging, and Git-based archive sharing (publish/subscribe). It is designed for privacy-conscious users who want offline access to their Slack history without cloud dependency. The architecture follows a clean internal package separation: `cmd/slacrawl` is the entry point, `internal/cli` handles command routing via Kong, `internal/store` provides the SQLite persistence layer with schema migrations and batch writes, `internal/syncer` orchestrates multi-source synchronization, and `internal/slackapi`, `internal/slackdesktop`, `internal/slackmcp`, and `internal/provider` implement source-specific adapters. The codebase is well-tested with extensive Go tests, CI/CD via GitHub Actions, and cross-platform releases (Linux, macOS, Docker). While primarily an application, slacrawl exhibits data pipeline characteristics: it ingests raw Slack data, transforms it into a normalized relational schema with FTS indexing, and exposes analytical outputs (reports, digests). The embedding_jobs table hints at future ML integration. The Git archive sharing module is a novel pattern for distributed data replication. Overall, the codebase demonstrates strong engineering practices with reusable components like the generic MCP client, platform-specific file locking, and the external provider protocol.
This repository is a data pipeline that aggregates and consolidates hosts files from 20+ upstream sources (adaway.org, mvps.org, someonewhocares.org, yoyo.org, URLHaus, KADhosts, hostsVN, and others) into unified blocklists for adware, malware, tracking, and optional categories (porn, gambling, fake news, social media). The core pipeline (`updateHostsFile.py`) fetches remote hosts files via HTTP, parses them, deduplicates entries, applies whitelist/blacklist filters, and generates a primary `hosts` file plus 30+ alternate permutations in `alternates/`. Each source has metadata in `update.json` (URL, license, frequency). The `makeHosts.py` script orchestrates full matrix generation of all extension combinations. Output includes stats tracking (`stats.out`, `stats.png`) and auto-generated READMEs via `updateReadme.py`. The repo includes Docker support, GitHub Actions CI (lint, test, multi-Python), CodeQL scanning, and automated dependency updates via Dependabot. Data sources are external URLs; the pipeline is idempotent and designed for scheduled runs.
This repository is a community-maintained domain classification dataset used by V2Ray/Project V for geosite routing. It contains over 1,500 plaintext list files organized by company, service, region, or functional category (e.g., ads, AI, education, porn). Each file uses a compact rule language supporting domain, full, keyword, regexp entries, attributes (@cn, @!cn, @ads), and include directives for hierarchical composition. A Go builder (main.go) compiles these lists into a protobuf binary (dlc.dat) with deduplication, redundant subdomain pruning, and optional custom profile assembly. A companion tool (cmd/datdump) reverses the binary back to plaintext for auditing. The repository is primarily a data pipeline: raw curated lists are transformed into a consumable routing artifact. While the Go tooling is essential, the core value lies in the curated data and its modular, attribute-based taxonomy. This hybrid nature (data with application tooling) is typical of domain-specific datasets that require compilation.
This repository is a community-driven web application that aggregates and displays developer conferences worldwide, with a strong focus on Call for Papers (CFP) deadlines. The core data is maintained in markdown files (README.md for current year, archives/ for past years) and enriched with CSV metadata (attendee counts, talk durations, AI-generated tags). A React single-page application (Vite, React Router, Leaflet) provides multiple views: calendar, list, map, CFP deadline tracker, and a CFP companion for speaker status tracking. The frontend features i18n (5 languages), theme toggle, favorites, offline support via service worker, and a sophisticated filtering system with include/exclude modes and URL-driven state. The backend consists of Node.js scripts that parse markdown, generate JSON feeds, ICS calendars, RSS, geocode locations, and validate data. AI integration (OVHcloud AI Endpoints) generates tags and attendee estimates. The project is actively maintained with extensive CI/CD (GitHub Actions for validation, geocoding checks, tag generation, security scanning) and includes unit tests (Vitest), E2E tests (Playwright), and BDD specs (Cucumber). While primarily an application, it has a significant data pipeline component, making it a hybrid project. The codebase is well-structured with reusable components like FilterMultiSelect and custom hooks, though some duplication exists in event forms and the markdown data format is fragile.
This is a production-grade Customer Data Platform (CDP) backend written in Go. It ingests events via HTTP (gateway), transforms and enriches them (processor), and routes them to over 90 destinations including streaming services (e.g., Kafka, Kinesis) and data warehouses (e.g., Snowflake, BigQuery, ClickHouse). The core of the system is a PostgreSQL-backed job queue (JobsDB) that provides partitioned, multi-consumer, durable processing. The architecture is a monolith with multiple internal services that can be deployed in different modes (gateway-only, processor-only, embedded). It includes enterprise features like OAuth token management, user suppression, regulatory deletion (GDPR), and reporting with HyperLogLog aggregation. The codebase is well-structured with clear separation of concerns, extensive unit and integration tests, and many reusable patterns (worker pool, circuit breaker, transaction listeners, config reloading). It is actively maintained with regular commits and security patching.
Star History is a web application that visualizes GitHub repository star growth over time through interactive charts and embeddable SVG badges. The system consists of a Next.js 14 frontend with D3-based chart rendering, a Hono backend that generates SVG charts and Open Graph cards on demand, and a shared code library used by both. A data pipeline (gh/) fetches repository metadata from the GitHub API and weekly activity stats from BigQuery (GitHub Archive), stores them in SQLite, and exports JSON files consumed by the frontend for leaderboards, weekly rankings, and star count pyramids. The application also includes a blog with curated open-source project roundups, a sponsor system, and a Chrome extension for quick chart viewing. Architecturally, the frontend uses URL hash-based state management to persist chart configurations (repos, timeline mode, log scale) and supports embedding via iframe or SVG. The backend caches star data, SVG charts, and OG cards using LRU caches with TTL and size limits, and rotates GitHub API tokens with cooldown to handle rate limits. The shared chart utilities (axis drawing, labels, legend, watermark) are used both server-side (via JSDOM) and client-side, demonstrating a clean separation of rendering logic. External dependencies include the GitHub REST API, BigQuery for weekly stats, GH Archive for event data, D3.js for chart rendering, JSDOM and Satori for server-side SVG/OG card generation, and SVGO for SVG optimization. The project is deployed with Docker containers (frontend on Cloudflare Pages, backend on GKE) and uses GitHub Actions for weekly data refreshes and deployments. While primarily an application, the repo has a strong data pipeline component that ingests, transforms, and exports structured data about GitHub repositories. This hybrid nature is evident in the gh/ directory, which functions as an ETL pipeline, and the blog content that analyzes open-source trends using star history data.
This repository is an IPTV source aggregation and serving application. It fetches M3U playlists and EPG XML data from multiple upstream providers (e.g., epg.pw, iptv-org, fanmingming/live, YueChan/Live, etc.), applies filtering, deduplication, and format conversion, and outputs cleaned playlists, text-format channel lists, source metadata, and TV-box-compatible JSON files. The system includes a Koa-based web server that serves the generated files and provides an API for channel availability checking via an optional iptv-checker service. It is designed to run as a scheduled GitHub Action (every 2 hours) or as a Docker container, with a rollback mechanism that falls back to mirror sites if primary sources fail. The codebase is well-structured with clear separation of concerns, custom rule support for channel inclusion/exclusion and URL replacement, and comprehensive file output management. It is a monolith application with a data pipeline character, primarily targeting IPTV enthusiasts and self-hosters.
Jitsu is an open-source event data platform (alternative to Segment) that ingests events from websites, apps, and servers, transforms them via user-defined functions (UDFs) and built-in destination functions, and delivers them to data warehouses (ClickHouse, BigQuery, Snowflake, Redshift, Postgres, S3, GCS) and SaaS tools (Amplitude, PostHog, HubSpot, etc.). The architecture is multi-service: a Go-based bulker handles ingestion and warehouse delivery, a TypeScript-based rotor executes event transformations and profile building, a Next.js console provides the management UI and API, and a CLI tool supports project management and function deployment. The data flow is: HTTP ingest → Kafka → rotor (functions) → bulker (SQL/file destinations) → ClickHouse/PostgreSQL for metrics and logs. External dependencies include Kafka, PostgreSQL, ClickHouse, Redis, MongoDB, and Google Cloud Storage. The console supports multi-tenant workspaces, authentication (Firebase, OIDC, NextAuth), billing, audit logging, rate limiting, and an MCP server for AI tool integration. The codebase is a monorepo with pnpm workspaces and Go modules, featuring extensive test suites (unit, integration with testcontainers, e2e) and CI/CD pipelines. It demonstrates strong engineering practices with reusable libraries (juava, jitsubase), custom high-performance JSON parsers, and well-abstracted repository and configuration patterns. However, there is some technical drift: duplicate JSON libraries (jsoniter and jsonorder), commented-out code, and ESLint disable comments in admin routes. Overall, it is a mature, actively maintained platform suitable for production use.
This repository is a curated collection of domain blocklists for various categories (ads, malware, porn, etc.) with automated build and maintenance pipelines. It ingests raw domain lists from community submissions and upstream sources, normalizes them, and outputs them in multiple formats (plain text, AdGuard, dnsmasq, etc.) suitable for use in DNS-level content filtering. The project includes a Python package for list processing (normalization, validation, merging), scripts for triaging issues, and GitHub Actions workflows for continuous integration and release management. It serves as a central resource for users of Pi-hole, AdGuard Home, and similar DNS filtering tools. The codebase is well-structured with tests and CI, though the core logic is tightly coupled to the project's configuration and domain-specific rules.
Feldera is a real-time streaming SQL analytics platform built on the DBSP (Differential Dataflow) incremental computation engine. It compiles SQL into efficient incremental circuits that process streaming data with sub-millisecond latency, supporting complex queries including joins, aggregations, window functions, and recursive queries. The system is deployed as a pipeline manager (Rust) with a REST API, a web console (Svelte), a Python SDK, a dbt adapter, and a CLI tool. It integrates with numerous data sources and sinks (Kafka, Postgres, Delta Lake, Iceberg, HTTP, NATS, Redis, etc.) and provides fault tolerance via checkpointing and storage backends. The architecture is polyglot: a Java-based SQL compiler (using Apache Calcite) translates SQL to Rust code that runs on the DBSP runtime; the pipeline manager orchestrates compilation, deployment, and lifecycle; the web console offers a full UI for pipeline management, monitoring, and profiling. The codebase also includes a Spark SQL translator (felderize) and extensive benchmark suites (Nexmark, LDBC). While primarily an application, it has strong research roots in incremental view maintenance and formal monotonicity analysis, reflected in the compiler's optimization passes and the DBSP runtime's operator design. Key reusable components include the adapter library (connector traits), the Feldera types crate (shared configs and errors), the common UI component library, and the Python client SDK. The project is actively maintained with frequent commits, comprehensive CI/CD, and a growing test suite covering SQL semantics across many data types and edge cases.
This repository implements a curated directory of Chinese-language Telegram channels, groups, and bots. It combines a Python-based data pipeline (scraping, filtering, categorization, and README generation) with a modern Astro static site that provides search, category browsing, and PWA features. The system ingests links from multiple sources (existing README, tg-nav, GitHub Issues), applies strict filtering rules (language, harmful keywords, subscriber thresholds), and outputs both a human-readable README and a structured JSON consumed by the frontend. The architecture is a hybrid of data engineering and web application, with the data pipeline being the core value proposition. The frontend is well-crafted with SEO, analytics, offline support, and a clean UI. The codebase demonstrates solid engineering practices, including reusable filtering logic, component-based UI, and automated build steps.
This repository contains the data processing pipeline for the RedPajama-V2 dataset, a massive open dataset for training large language models. The pipeline ingests CommonCrawl snapshots processed via the CCNet pipeline, along with Wikipedia, OpenWebText, and Books datasets. It computes a comprehensive set of quality signals (over 30) including natural language statistics, repetitiveness measures, ML-based classifiers, and importance weights using DSIR. Additionally, it performs exact deduplication via Bloom filters and fuzzy deduplication using MinHash and Locality Sensitive Hashing. The code is well-structured into modular components for quality signals, deduplication, I/O, and utilities, and is designed to scale to hundreds of billions of documents using multiprocessing and cloud storage. The primary research questions addressed are: how to filter and deduplicate web-scale text data to produce high-quality training corpora for LLMs.
OpenMoji is an open-source emoji set maintained by HfG Schwäbisch Gmünd. The repository contains the full pipeline for producing emoji assets: source SVGs, a metadata generation system that merges emojibase-data with custom CSV enhancements and Unicode draft overrides, export scripts for color/black SVGs, PNGs, PDFs, and font generation via Docker. The central data file `data/openmoji.json` provides structured emoji metadata including hexcodes, tags, authors, and dates. The build system is modular with incremental caching, color palette enforcement using kd-tree, and comprehensive tests for data integrity, SVG layer rules, and font table presence. While primarily a data/research repository producing emoji assets, it includes a thin npm package for programmatic access and a sophisticated build pipeline that could be adapted for other icon sets. The repository is actively maintained with recent commits and CI workflows, though it relies on external dependencies like emojibase-data and a Docker font builder that may introduce drift.