Polars is a high-performance DataFrame library and query engine written in Rust with Python bindings. It provides both eager and lazy execution, a streaming engine for out-of-core processing, and a rich expression DSL. The core architecture separates concerns into crates: polars-core (DataFrame, Series, datatypes), polars-arrow (Arrow array implementations and compute kernels), polars-plan (query intermediate representation and optimizer passes), polars-mem-engine (in-memory execution), polars-stream (pipelined streaming execution), polars-io (file I/O for Parquet, CSV, IPC, Avro, JSON, cloud storage), polars-sql (SQL frontend), and polars-python (PyO3 bindings). Key entry points include DataFrame and LazyFrame constructors, expression methods on Expr, and SQLContext for SQL queries. Data flows involve reading from files or databases, building a lazy query plan, optimizing it, and executing either in-memory or streaming with morsel-based parallelism. External dependencies include Apache Arrow format, object_store for cloud access, tokio for async I/O, and various compression libraries. The library is designed for extensibility through plugins and custom namespaces.
Pathway is a high-performance live data processing framework built on top of timely dataflow and differential dataflow, providing a Python API for defining streaming and batch data pipelines. It is designed for real-time ETL, event processing, AI/ML pipelines (including RAG), and incremental computation with strong consistency guarantees. The architecture consists of a Rust core engine that handles dataflow execution, scheduling, persistence, and connectors, while the Python layer provides a declarative API for table operations, UDFs, and integrations with external systems like Kafka, databases, vector stores, and cloud storage. Key entry points include the `pathway` CLI, the Python `pw` module, and the Rust engine library. Data flows through connectors (sources/sinks), transformation operators (joins, groupby, windowing, etc.), and optional persistence for fault tolerance. External dependencies include Kafka, S3, PostgreSQL, various vector databases, and LLM services.
Apache Airflow is a workflow orchestration platform for authoring, scheduling, and monitoring data pipelines. The system uses a modular architecture with core components, extensible providers, a React/TypeScript web UI, and a FastAPI-based REST API. Key entry points include the CLI (`airflow` command), web server, scheduler, and triggerer. Data flows involve DAG parsing, task scheduling via executors, and state persistence in a metadata database. External dependencies include PostgreSQL/MySQL, Redis/RabbitMQ for Celery, and various provider-specific integrations. The codebase demonstrates strong engineering practices with version compatibility patterns, plugin systems, and well-structured API layers.
RocketRide is a high-performance data processing engine with modular nodes, AI capabilities, and cross-platform clients. The system provides a visual interface for building data pipelines that integrate various data sources, AI/ML services, and processing nodes. At its core, it features a C++-based execution engine for performance, Python-based processing nodes for flexibility, and multiple client applications (web, desktop, VS Code extension) for interaction. The platform emphasizes extensibility through its node system, allowing users to create custom processing components while providing built-in integrations for databases, cloud storage, AI services, and popular APIs. It includes monitoring, observability, and deployment capabilities for production use.
Miller (mlr) is a command-line data processing tool for structured text formats including CSV, TSV, JSON, JSON Lines, YAML, DKVP, and others. It operates as a streaming pipeline: input records are read from files or stdin, processed through a chain of transformer verbs (e.g., cut, sort, join, stats1, put, filter), and written to output in any supported format. The tool includes a domain-specific language (DSL) for record-level transformations, with support for user-defined functions, subroutines, loops, and higher-order functions. Miller is designed for performance, using Go channels for concurrent streaming, batch processing, and a slab allocator for efficient memory use. The architecture is a monolithic CLI application written in Go with no external runtime dependencies. Key entry points include the main binary (cmd/mlr/main.go), the entrypoint dispatcher (pkg/entrypoint), and the terminal subcommands (help, repl, script, regtest, mcp). The codebase is organized into packages for input readers, output writers, transformers (verbs), DSL parsing and execution, CLI flag handling, and a core value type system (Mlrval). Miller also provides an MCP (Model Context Protocol) server for AI agent integration, shell completion scripts, and a REPL for interactive use. Miller is not a research project; it is a production-grade data munging tool comparable to awk, sed, cut, join, and sort but for name-indexed data. Its primary data flows involve reading heterogeneous records, applying transformations via verbs or DSL expressions, and writing results. The tool handles compressed files, in-place editing, and output redirection to files or pipes.
Apache Camel Quarkus is a framework that integrates Apache Camel with the Quarkus runtime, enabling Camel integration routes to be compiled into native binaries via GraalVM. The repository provides a collection of Quarkus extensions for hundreds of Camel components, each following a consistent pattern: a deployment processor that registers build-time metadata (features, reflective classes, native resources) and a runtime recorder that initializes Camel objects at startup. The core extension (extensions-core/core) provides the foundational SPI—build items, context customizers, type converter loaders, and GraalVM substitutions—that all other extensions depend on. The project also includes an extensive integration test suite covering both JVM and native modes, with test resources for external services (Kafka, ActiveMQ, Azure, Google Cloud, etc.) and conditional test execution based on platform or environment. Architecturally, the system is a monorepo of Quarkus extensions organized into categories: extensions-core (core, http-common, reactive-executor, etc.), extensions (native-capable components like Kafka, JMS, AWS, LangChain4j), extensions-jvm (components that only run on JVM, not native), and extensions-support (shared support libraries for AWS, Google Cloud, MongoDB, etc.). Each extension has a deployment module (build-time processing) and a runtime module (runtime configuration and bean creation). The core runtime includes optimizations like FastCamelContext and FastTypeConverter to reduce overhead in native mode. Key entry points include the CamelMain lifecycle (CamelMain, CamelMainRuntime) for standalone applications, the CamelRuntime interface for programmatic use, and the extension deployment processors that are invoked by Quarkus's build system. Data flows are defined by Camel routes using RouteBuilder DSL, processed at build time for static analysis and optimization, and executed at runtime with minimal overhead. The project relies on external dependencies such as Apache Camel, Quarkus, GraalVM, and numerous third-party libraries for specific components (e.g., AWS SDK, Azure SDK, Kafka clients). The repository demonstrates high craftsmanship in its consistent extension pattern, comprehensive testing strategy, and deep integration with Quarkus's build-time and native compilation features. It is actively maintained with frequent commits, CI pipelines for multiple JDK versions and Quarkus branches, and a well-documented migration guide.
This repository is a community-maintained, curated registry of EVM-compatible blockchain networks. It contains thousands of JSON files under `_data/chains/`, each defining a chain's metadata: RPC endpoints, native currency, explorers, icons (IPFS CIDs), status (active/deprecated/incubating), parent L2 relationships, and feature flags. The dataset is the primary asset, designed for consumption by wallets, dApps, block explorers, and infrastructure tooling that needs up-to-date chain information. Beyond the data, the repository includes a Kotlin-based processor that validates all chain definitions (schema compliance, icon size/format, IPFS retrievability, ENS address checks) and produces aggregated `chains.json` and `chains_mini.json` outputs. A published Kotlin model library (`model/`) and an HTTP loader (`httpsloader/`) enable programmatic access. CI pipelines enforce formatting, schema validation, and per-PR checks. The project is hybrid: its core value is the dataset, but it also delivers a well-engineered application layer for validation, aggregation, and distribution. The registry answers questions like 'What RPC endpoints are available for chain X?', 'What is the native currency of chain Y?', 'Which chains are L2s of Ethereum?', and 'What explorers support EIP3091?'. It serves as a single source of truth for EVM chain metadata, reducing fragmentation across the ecosystem.
The Data Engineering Wiki is a community-maintained knowledge base dedicated to documenting data engineering concepts, tools, and best practices. Rather than being an executable application or data processing pipeline, it serves as a comprehensive reference resource for individuals learning and working in the data engineering field. The repository is organized into thematic sections covering foundational concepts (data architecture, modeling, processing, storage), practical tools and technologies, implementation guides, frequently asked questions, and curated learning resources. Structurally, the wiki follows a logical progression from theoretical foundations to practical application, with concepts explained through clear definitions, diagrams, and real-world examples. Key strengths include its comprehensive coverage of both established and emerging data engineering topics, practical guides on data pipeline best practices and testing methodologies, and active maintenance evidenced by recent commits. The content is presented in an accessible markdown format with consistent frontmatter, making it easy to navigate and consume. While not containing executable code or data pipelines itself, the wiki provides reusable knowledge assets in the form of methodological guides and reference materials that can be directly applied to data engineering projects. Notably, the Data Pipeline Best Practices and Testing Your Data Pipeline guides offer actionable frameworks for building reliable, scalable data systems. The repository's value lies in its role as a living document that captures and organizes community knowledge to support learning and professional practice in data engineering.
AIStore (AIS) is a high-performance, distributed object storage system engineered for AI/ML workloads. It operates as a cluster of proxy (gateway) and target (storage) nodes, providing S3-compatible API, multi-cloud backend support (AWS, GCP, Azure, OCI), and advanced data management features such as erasure coding, mirroring, LRU caching, and distributed resharding (dsort). The system is designed for linear scale-out and consistent performance, with a rich set of extended actions (xactions) for batch operations like copy-bucket, archive, blob download, and ETL transformations. It includes a full-featured CLI (`ais`), a Python SDK with PyTorch dataset integration, and a standalone authentication server (AuthN). The codebase is primarily a production-grade application with strong software engineering practices: modular architecture, extensive unit and integration tests, code generation for high-performance serialization (msgp), and build-tag-based feature gating. It demonstrates non-obvious craftsmanship in areas like memory management (slab allocators, SGL), intra-cluster transport (stream bundles), and probabilistic data structures (dynamic cuckoo filters). While the core is a storage system, it has significant data/research features (ETL pipelines, distributed sorting, shard indexing) that make it a hybrid tool for AI data pipelines. Key entry points include `cmd/aisnode/main.go` (server), `cmd/cli/main.go` (CLI), `cmd/authn/main.go` (authentication), and `python/aistore/sdk/` (Python SDK). The system relies on cloud SDKs, Kubernetes client, and various Go libraries. It is actively maintained with frequent commits, comprehensive CI/CD, and a large test suite.
Flowcraft is a workflow orchestration framework for building, executing, and monitoring durable workflows. It provides a type-safe, composable API for defining workflows as directed graphs of nodes and edges, supporting both programmatic (fluent) and declarative (JSON) definitions. The framework enables progressive scaling from in-memory execution to distributed systems via adapters for message queues and cloud services, with built-in support for patterns like retries, timeouts, human-in-the-loop, batch processing, and subflows. Key architectural pieces include a dependency-free core runtime, a sophisticated context tracking system for efficient state persistence, a plugin-based adapter architecture for distributed execution, and LLM tooling for workflow introspection and control via natural language.
This repository is a data pipeline that aggregates and consolidates hosts files from 20+ upstream sources (adaway.org, mvps.org, someonewhocares.org, yoyo.org, URLHaus, KADhosts, hostsVN, and others) into unified blocklists for adware, malware, tracking, and optional categories (porn, gambling, fake news, social media). The core pipeline (`updateHostsFile.py`) fetches remote hosts files via HTTP, parses them, deduplicates entries, applies whitelist/blacklist filters, and generates a primary `hosts` file plus 30+ alternate permutations in `alternates/`. Each source has metadata in `update.json` (URL, license, frequency). The `makeHosts.py` script orchestrates full matrix generation of all extension combinations. Output includes stats tracking (`stats.out`, `stats.png`) and auto-generated READMEs via `updateReadme.py`. The repo includes Docker support, GitHub Actions CI (lint, test, multi-Python), CodeQL scanning, and automated dependency updates via Dependabot. Data sources are external URLs; the pipeline is idempotent and designed for scheduled runs.
This is a production-grade Customer Data Platform (CDP) backend written in Go. It ingests events via HTTP (gateway), transforms and enriches them (processor), and routes them to over 90 destinations including streaming services (e.g., Kafka, Kinesis) and data warehouses (e.g., Snowflake, BigQuery, ClickHouse). The core of the system is a PostgreSQL-backed job queue (JobsDB) that provides partitioned, multi-consumer, durable processing. The architecture is a monolith with multiple internal services that can be deployed in different modes (gateway-only, processor-only, embedded). It includes enterprise features like OAuth token management, user suppression, regulatory deletion (GDPR), and reporting with HyperLogLog aggregation. The codebase is well-structured with clear separation of concerns, extensive unit and integration tests, and many reusable patterns (worker pool, circuit breaker, transaction listeners, config reloading). It is actively maintained with regular commits and security patching.
This repository is a Ruby gem (polars-df) that provides a high-performance DataFrame library by binding to the Rust-based Polars engine. It serves as a native extension, allowing Ruby developers to perform ETL, data transformation, and analytics tasks with Rust-level speed while maintaining Ruby syntax. The architecture consists of a Rust core (ext/polars) handling computation and memory management, wrapped by Ruby modules (lib/polars) that expose a fluent DSL for data manipulation. It supports both eager (DataFrame/Series) and lazy (LazyFrame/Expr) evaluation modes, with extensive I/O capabilities for formats like CSV, Parquet, and Arrow, as well as cloud storage and database integration. The codebase demonstrates strong engineering practices, utilizing the `magnus` crate for safe Ruby-Rust FFI and implementing complex type conversion logic to bridge the two languages. It includes comprehensive test suites and CI pipelines for cross-platform compilation. However, the tight coupling to the upstream Polars Rust library introduces maintenance overhead, as breaking changes in the Rust core require synchronous updates to the Ruby bindings. Several features, particularly around cloud catalogs (Iceberg, Delta) and advanced selectors, are marked as experimental or contain incomplete implementations (Todo blocks), indicating an active but evolving codebase. For AI agents and engineers, this library is the primary tool for data engineering within the Ruby ecosystem where performance is critical. It is suitable for building data pipelines, analytics services, or backend systems that require efficient data processing without leaving the Ruby environment. The presence of `unstable` markers and `Todo` blocks suggests that while the core functionality is robust, some advanced features should be used with caution in production environments.
CourtListener is a comprehensive legal research platform operated by Free Law Project. It aggregates U.S. court opinions, dockets, oral arguments, and judge profiles from multiple sources including PACER/RECAP, Harvard Caselaw, Columbia, state courts (Florida, Texas, New York), and SCOTUS. The system provides full-text search via Elasticsearch, real-time and scheduled alerts, REST API with OAuth2, webhook notifications, citation analysis, and audio transcription. It also includes tools for bulk data import, data cleaning, and embedding generation for LLM training. Architecturally, it is a Django 6.0+ monolith with a Celery task queue (Redis broker), PostgreSQL database, Elasticsearch search engine, and S3-compatible storage. The frontend uses Django templates with Alpine.js and some React components, built via Webpack. The system runs as multiple Docker containers (web, Celery workers, scrapers, daemons). Key data flows include nightly PACER scraping, email-based RECAP document ingestion, citation extraction using eyecite, and state-specific docket merging pipelines. The system is heavily event-driven with pgtrigger-based audit trails and Redis-based throttling. While primarily an application, it has a strong data research character: extensive ETL pipelines, machine learning integration (LLMs for text extraction, OpenAI Whisper for transcription), citation network analysis, and state court data modeling. The codebase demonstrates high craftsmanship in reusable patterns like the CeleryThrottle, tiered caching, and generic merger classes, but also carries some technical debt from many SQL migrations and tightly coupled scrapers.
dataset is a lightweight Python library that simplifies database operations by providing a dict-like interface on top of SQLAlchemy. It automatically creates tables and columns on insert, supports CRUD operations, transactions, and chunked bulk inserts/updates. The library is designed for small to medium-scale data tasks where full ORM overhead is unnecessary. It supports SQLite, PostgreSQL, and MySQL, and is fully type-annotated with mypy strict compliance. The codebase is well-organized into core modules (database, table, types, util, chunked) and has thorough test coverage across all supported backends. While the core abstraction is thin (intentionally), the implementation demonstrates solid engineering discipline with clean APIs, thread safety considerations, and proper handling of SQLAlchemy 2.x quirks.
OpenDBT is a Python library and CLI tool that extends dbt-core functionality through runtime patching. It provides enhanced capabilities including custom adapter integration, local Python model execution, DLT job integration, Airflow orchestration, multi-project (DBT Mesh) support, and an advanced documentation UI with column-level lineage. The architecture uses runtime patching to modify dbt's behavior without changing its source code, featuring a modular design with version-specific adaptations for dbt v1.7 and v1.8+, shared components for cross-version functionality, and dedicated integrations for Airflow and catalog generation. Entry points include the 'opendbt' CLI command, OpenDbtProject Python API, and Airflow operators. Data flows involve user CLI commands triggering enhanced dbt executions, documentation generation creating catalogl.json with lineage data, and Airflow operators orchestrating dbt tasks within DAGs.
This repository is a large-scale, community-curated collection of free IPTV (Internet Protocol Television) live stream URLs, organized as standardized M3U playlist files by country and platform (e.g., Pluto TV, Samsung TV Plus, local broadcasters). The project functions as a data pipeline: raw stream URLs are contributed via GitHub issues, validated, formatted, and aggregated into categorized playlists (by country, language, category, region, and source) which are deployed to GitHub Pages and a JSON API. Supporting TypeScript scripts handle playlist parsing, stream testing, metadata enrichment from an external database (iptv-org/database), and automated updates via GitHub Actions. While the primary deliverable is the curated dataset, the repository includes a well-structured automation framework that could be adapted for similar data aggregation tasks. The codebase is actively maintained with daily workflows, though the underlying stream URLs are inherently unstable and subject to geo-blocking or link rot.
This repository is a curated collection of domain blocklists for various categories (ads, malware, porn, etc.) with automated build and maintenance pipelines. It ingests raw domain lists from community submissions and upstream sources, normalizes them, and outputs them in multiple formats (plain text, AdGuard, dnsmasq, etc.) suitable for use in DNS-level content filtering. The project includes a Python package for list processing (normalization, validation, merging), scripts for triaging issues, and GitHub Actions workflows for continuous integration and release management. It serves as a central resource for users of Pi-hole, AdGuard Home, and similar DNS filtering tools. The codebase is well-structured with tests and CI, though the core logic is tightly coupled to the project's configuration and domain-specific rules.
Dataform is an open-source data transformation and workflow orchestration tool focused on BigQuery. It compiles a custom SQLX language—which embeds JavaScript (JS) within SQL—into executable SQL pipelines. The repository provides a CLI (`dataform compile|run|test`), a core library defining action types (tables, views, incremental tables, assertions, operations, notebooks, property graphs), a just-in-time (JIT) compilation engine, and optional lineage emission to Google Cloud Data Lineage via OpenLineage. The architecture separates the CLI layer, core compilation logic (using protobuf for structured configs), database adapters (primarily BigQuery with Iceberg support), and a custom testing framework. Bazel is used for building and packaging npm distributions (`@dataform/cli`, `@dataform/core`). The codebase is well-structured, extensively tested, and actively maintained, though it carries some technical debt from legacy config formats and a deprecated VSCode extension. Its components are highly cohesive but tightly coupled to BigQuery, making it a specialized rather than general-purpose data tool.
This is the DataTalksClub Data Engineering Zoomcamp — a free, instructor-led 9-week educational course repository, not a deployable application. It is a hybrid collection of teaching materials: Jupyter notebooks, dlt pipeline scripts, dbt models with tests, Airflow/Kestra/Spark/streaming examples, and workshop content. The dominant character is data_research: the repository exists to teach end-to-end data engineering through hands-on pipelines over NYC taxi trip data (yellow/green/FHV in CSV/Parquet from GCS) and the Open Library API. Data flows from raw sources through dlt extraction/normalization/loading into DuckDB/BigQuery, then through dbt transformations (staging/core layers) with singular/generic/unit tests and model contracts, plus batch processing with Spark and streaming with Kafka. The course is actively maintained with cohorts from 2022 through 2027, and includes a systematic image-distrust audit that removed 64 AI-generated illustrations from lessons.