This is the DataTalksClub Data Engineering Zoomcamp — a free, instructor-led 9-week educational course repository, not a deployable application. It is a hybrid collection of teaching materials: Jupyter notebooks, dlt pipeline scripts, dbt models with tests, Airflow/Kestra/Spark/streaming examples, and workshop content. The dominant character is data_research: the repository exists to teach end-to-end data engineering through hands-on pipelines over NYC taxi trip data (yellow/green/FHV in CSV/Parquet from GCS) and the Open Library API. Data flows from raw sources through dlt extraction/normalization/loading into DuckDB/BigQuery, then through dbt transformations (staging/core layers) with singular/generic/unit tests and model contracts, plus batch processing with Spark and streaming with Kafka. The course is actively maintained with cohorts from 2022 through 2027, and includes a systematic image-distrust audit that removed 64 AI-generated illustrations from lessons.
GQLAlchemy is a Python library and Object Graph Mapper (OGM) for the Memgraph graph database, with secondary support for Neo4j. It provides a fluent Cypher query builder, declarative node/relationship models (Pydantic v2-based), and full lifecycle management of graph database instances (Docker/binary), streams (Kafka, Pulsar), triggers, and high-availability routing. The library also includes a rich set of transformations for bidirectional conversion between Memgraph/Neo4j and popular graph machine learning frameworks (NetworkX, DGL, PyG, TF-GNN), as well as tabular data import (CSV, Parquet, ORC, Feather) from local filesystem, S3, and Azure Blob Storage. While primarily a software library for building graph applications, it has strong data research capabilities through its graph ML translators and ETL importers, making it a hybrid tool suited for both production graph apps and data science pipelines. The codebase is well-tested (unit and integration tests), has comprehensive CI/CD, and follows a modular architecture with abstract base classes for extensibility. However, some modules are large (e.g., monolithic query builders), and optional dependencies (PyTorch, DGL, TF-GNN, PyArrow) increase maintenance surface, with a medium level of technical drift due to some deprecated Pydantic v1 patterns and skipped tests.
The ReplicaDB repository is a high-performance, open-source command-line tool for bulk data replication between heterogeneous databases. It supports a wide range of data sources, including relational databases, NoSQL databases, data virtualization platforms, file formats, cloud storage, and streaming platforms. The tool is cross-platform compatible with Windows, Linux, and macOS, and leverages parallel data transfer for optimal performance and system utilization during large-scale data migrations and synchronization tasks.
This is a curated content hub for Shadowrocket, an iOS network proxy app, hybrid in nature: its user-facing value is a Chinese-language README catalog of installable resources (original .sgmodule modules for ad-blocking, traffic-splitting policies for DeepSeek/Emby/Wi-Fi Calling, DNS overrides for CMCC IPTV; .list rule sets; a lazy configuration; 30+ original color themes encoded as shadowrocket://color URL schemes), while its only executable surface is a scheduled data-aggregation pipeline. The live GitHub Pages site referenced throughout the README (redirect.html, theme.html, images) is not present in this branch's file tree — only README.md, a generated modules.json, a CI workflow, and one build script are visible, so the repo-as-code is essentially an ETL job, not a runtime service. The functional core is scripts/build-modules.js, a dependency-free Node.js script executed by GitHub Actions every 6 hours. It parses the repo's own README for .sgmodule/.srmodule/.module URLs, enumerates module files from community repos (fmz200/wool_scripts, zirawell/R-Store) via the GitHub Trees API, probes each module's first 2KB using HTTP Range requests to extract #! metadata (name, description, icon), enriches entries with icons resolved from three icon libraries using Chinese/English app-alias matching, deduplicates and pins key tools (Script-Hub, Sub-Store, BoxJs), then writes modules.json — the feed consumed by the ShadowStore module directory (shadowstore.dpdns.org). External dependencies are GitHub API endpoints, raw.githubusercontent.com, and a GITHUB_TOKEN for API rate limits; there is no package.json or npm dependency graph. There are no unit tests and the script is tuned to Shadowrocket/Surge file formats, but it demonstrates transferable engineering patterns: github.com/blob to raw.githubusercontent.com URL normalization, partial-content fetch for cheap metadata extraction, 404/invalid-content detection, a bounded-concurrency worker pool with progress logging, and alias-based fuzzy icon resolution. Maintenance is genuinely active (automated syncs commit modules.json regularly; last commit 2026-09-04), and technical drift is low — Node 22 LTS, current GitHub Actions checkout/setup-node v5, modern JS syntax (global fetch, AbortController, nullish coalescing).
This repository provides curated Chinese character sets (3500 common, 7000 common, and a comprehensive set) combined with symbols (Latin letters, digits, punctuation, full-width variants). The data is sourced from official Chinese character frequency lists and community-maintained character tables. The repository includes simple Node.js scripts for building the final text files from source files, merging new characters, and removing duplicates. The primary use case is generating character sets for Unity TextMeshPro font creation, but the data can be used in any application requiring a comprehensive Chinese character inventory. The codebase is minimal and well-documented in AGENTS.md, with clear instructions for editing source files and regenerating outputs. The repository is actively maintained with no open issues and a recent commit.
Frappe Insights is a full-stack business intelligence application enabling users to connect to data sources, build queries visually or via SQL, create visualizations, and assemble dashboards and workbooks. The decoupled architecture features a Vue.js 3 frontend communicating with a Python/Frappe backend through REST APIs. Key capabilities include a multi-database query builder using Ibis and SQLGlot, ECharts-powered visualizations, role-based access control, and DuckDB-based data warehousing for caching imported tables. The frontend leverages Vue composition API for reusable logic, while the backend uses Frappe's DocType system for extensibility. The project demonstrates solid engineering practices with modular API design, reusable components, and abstraction layers, though complexity arises from supporting numerous database types and deep Frappe integration.
This repository is the official Kubernetes performance testing suite, housing multiple tools for benchmarking, load testing, and monitoring Kubernetes clusters. The core tool is clusterloader2 (CL2), a powerful framework that deploys scalable workloads, measures key performance indicators (API responsiveness, pod startup latency, scheduling throughput), and validates SLOs. It uses a pluggable measurement system, tuning sets for load patterns, and a provider abstraction layer supporting various environments (GKE, EKS, AKS, Kind, Kubemark). Other tools include a metric comparison utility (benchmark), a historical performance dashboard (perfdash), a real-time SLO Prometheus exporter (slo-monitor), DNS and network performance benchmarks, and data ingestion pipelines (perflens). The codebase is predominantly Go with Python scripts for data processing and extensive Prometheus/Grafana configurations. While primarily an application, it incorporates data pipelines for metric ingestion and SLO analysis, making it a hybrid toolset for performance engineering.
Parsons is a Python library providing a unified interface to dozens of third-party services commonly used in progressive political campaigns and advocacy organizations. It acts as an ETL/integration toolkit, with a core `Table` abstraction (backed by petl) that normalizes data from APIs, databases, and file formats. The library includes connectors for voter file platforms (NGP VAN, PDI, TargetSmart, Catalist), fundraising (ActBlue, Donorbox, Braintree), communication (Twilio, Slack, Gmail), event management (MobilizeAmerica, ActionNetwork), and cloud storage (S3, GCS, Azure). It also provides database connectors (Postgres, Redshift, BigQuery, MySQL, SQLite) and utilities for dbt integration, credential management, and file handling. The architecture is a monolith of loosely coupled connector modules, each following a consistent pattern of credential resolution via environment variables and returning `Table` objects. The codebase is well-tested with unit tests (mocked) and some integration tests, but many connectors rely on external APIs that may drift. The library is actively maintained by The Movement Cooperative, with recent commits and CI/CD pipelines.
Mage is an open-source data pipeline orchestration platform that combines a visual notebook-style editor with a production-grade scheduling and execution engine. The system enables users to build, run, and monitor data pipelines composed of Python, SQL, R, and dbt blocks. Architecturally, it is a full-stack web application with a Python backend (Tornado-based server, SQLAlchemy ORM, REST API) and a Next.js/React frontend with a custom component library. Key entry points include the CLI (`mage` command), the web UI served by `mage_ai/server/server.py`, and the API layer at `mage_ai/api/` which exposes CRUD resources for pipelines, blocks, runs, schedules, users, and integrations. Data flows center around pipeline execution: blocks are connected in a DAG, each block can be a data loader, transformer, or exporter. Execution is handled by a configurable executor (local, K8s, EMR, PySpark) and scheduled via a built-in scheduler or external triggers. The platform integrates with dozens of data sources and destinations through a Singer-based integration framework, supports streaming pipelines, and includes advanced features like RBAC, version control, AI-assisted pipeline generation, and a command center for global search. It depends heavily on external services such as cloud storage (S3, GCS, Azure), databases (PostgreSQL, BigQuery, Snowflake, etc.), and authentication providers (OAuth, LDAP, OIDC).
This repository is a curated collection of wordlists for bug bounty hunting and web application security testing. It aggregates URL paths, payloads, and vulnerability probes from multiple public sources (SecLists, OneListForAll, dirsearch, nuclei-templates, etc.) into a unified, organized dataset. The wordlists are categorized by technology (e.g., Apache, WordPress, Nginx) and severity (info, low, medium, high, critical), making them directly usable with tools like Nuclei, ffuf, and dirsearch. The repository includes two automation scripts: a shell script that fetches and deduplicates wordlists from external URLs, and a Go program that parses nuclei-templates YAML files to generate technology-specific wordlists. These scripts are run weekly via GitHub Actions to keep the dataset current. The repository is purely a data asset; it contains no application code, APIs, or services. Its primary value is as a ready-to-use, aggregated wordlist source for security researchers and penetration testers.
This repository is a data engineering pipeline that produces a comprehensive, unified country codes dataset (CSV/JSON) by merging information from multiple authoritative sources: ISO 3166, ISO 4217, UN Statistics, UN Protocol (UNTERM), CLDR, statoids, Geonames, EDGAR, FIFA, and Wikidata. The pipeline is orchestrated via a Makefile that sequentially runs Python scripts to fetch, parse, clean, and join data. The final output contains dozens of columns covering country names in multiple languages, ISO codes, currency codes, dialing codes, and more. The repository is primarily a data transformation project rather than a software application, though it includes sophisticated web scraping and fuzzy matching. The code is modular and well-structured, but reproducibility is hampered by a manual step requiring users to manually download and rename UNSD CSV files. The included GitHub Actions workflow automates monthly updates. Overall, the project is a solid reference data curation effort with moderate complexity.
The Multimodal Universe is a large-scale astronomical data repository that aggregates ~30 surveys (Legacy Survey, HSC, JWST, Gaia, SDSS, DESI, TESS, etc.) into a unified, HEALPix-partitioned HDF5 format, exposed through Hugging Face datasets. It covers multiple modalities: images (multi-band), spectra (1D), time series (light curves), and tabular catalogs. The repository provides dataset builder scripts (GeneratorBasedBuilder subclasses) per survey, cross-matching utilities, and benchmark training code (PyTorch Lightning) for downstream tasks like photometric redshift estimation and galaxy property regression. While primarily a data research project, it also includes application-like components such as baseline classifiers (Informer for PLAsTiCC) and experimental benchmark pipelines. The data is designed for training foundation models in astronomy, with streaming support and standardized column naming. Notable patterns include HEALPix-based spatial indexing, parallel processing in build scripts, and a consistent Hugging Face dataset interface across all surveys. The codebase is well-structured for its purpose, though many scripts are tightly coupled to specific surveys and external data sources, limiting direct reuse without adaptation.
This repository is an educational tutorial for Apache Spark using Scala, designed to teach distributed data processing concepts through practical examples. It functions as a data research and engineering learning resource, featuring Jupyter notebooks and Scala source code that demonstrate RDD operations, SparkSQL, and streaming pipelines. The project includes bundled datasets such as the King James Version Bible and the Enron Spam/Ham email corpus, along with golden output files for test verification. While it serves as a comprehensive guide for Spark development, it primarily focuses on teaching data transformation patterns rather than deploying a production service. The codebase demonstrates solid engineering practices for an educational context, including unit tests, custom utility classes for CLI argument parsing, and structured data schemas. However, the technical stack is significantly outdated, relying on Spark 2.3.0 and Scala 2.11.8, which are no longer supported. The examples frequently use deprecated APIs, such as the RDD-based DStream for streaming, which has been superseded by Structured Streaming in modern Spark versions. This creates a high technical drift, making the code less suitable for new projects without significant refactoring. Despite the outdated dependencies, the repository remains a valuable resource for understanding Spark fundamentals. The inclusion of golden files and test utilities provides a reproducible framework for validating data processing logic. The tutorial structure progresses logically from basic word counts to complex SQL and streaming operations, making it effective for learning. Users should be aware that while the patterns are sound, the specific API implementations will require updates to align with current Spark LTS versions.
This repository powers the personal finance blog 'OfDollarsAndData.com'. It is primarily a data research collection, consisting of hundreds of R scripts that import, clean, and analyze financial and economic datasets to produce publication-ready charts, tables, and interactive calculators for blog posts. The data lineage is clear: raw data is downloaded or read from external sources (Shiller S&P 500, FRED, BLS, Survey of Consumer Finances, YCharts, etc.) in import/ scripts, transformed and saved as RDS files in build/ scripts, and then consumed by analysis/ scripts that generate plots with a consistent ggplot theme. The repository also includes a webdev folder containing a WordPress theme with custom widgets and calculator embedding logic. While the codebase is not a packaged application and lacks testing or CI, the analysis is thorough and well-structured for its purpose. The hybrid nature includes both data research R scripts and web development PHP/JS code, but the dominant character is data research.
DBeam is a mature Java library and CLI tool that extracts data from SQL databases (PostgreSQL, MySQL, MariaDB, H2) and exports it as Apache Avro files, typically stored in Google Cloud Storage. It operates as a single-threaded Apache Beam pipeline, first inferring the table schema by reading one row, then streaming the full result set via JDBC directly into Avro binary encoding using a custom Beam sink. The tool supports partitioned exports, parallel query execution via a split column (experimental), multiple Avro codecs, KMS-decrypted passwords, and various array handling modes. It is designed for reliability in production batch data extraction pipelines and has been in use at Spotify since 2017. The codebase is well-tested and uses modern Java practices (AutoValue, Beam SDK 2.76), with CI/CD workflows for Maven Central releases.
This repository is a curated collection of Chinese vocabulary lists for standardized tests (HSK 3.0, BCT, YCT), enriched with pinyin, zhuyin, part-of-speech tags, frequency data, and English definitions. It includes Python and JavaScript scripts for data cleaning, format conversion (TXT/TSV/XLSX), POS tag correction using SUBTLEX frequency corpus, and pinyin tone conversion. The data is organized by test level and version (2021 and 2025 HSK standards), with multiple output formats suitable for flashcard apps (Anki, Pleco) and language learning tools. The repository is primarily a data asset with supporting ETL scripts, not a production application. It demonstrates solid data engineering practices for language resource curation, though it suffers from some file duplication and hardcoded paths.
This is the TigerGraph ecosystem repository, a large collection of tools, connectors, demos, benchmarks, and tutorials centered around the TigerGraph graph database platform. It is not a single application but an aggregation of community and in-house developed components. The dominant character is application: it includes production-grade ETL connectors (Spark DataSource V2, JDBC driver, Kafka Connect, Hadoop MapReduce), a multi-version GSQL CLI client, a Go-based REST API for remote GSQL management, a Kubernetes operator, and numerous demos showcasing graph analytics (PageRank, community detection, geospatial search, fraud detection, GraphRAG). There are also benchmarking suites (LDBC SNB) and sample data. The codebase exhibits high craftsmanship in the connector and client libraries, with robust retry logic, token management, streaming JSON parsing, and schema inference. However, it suffers from massive code duplication across GSQL client versions, incomplete JDBC driver implementations, hardcoded paths and credentials in demos, and outdated vendored libraries (Leaflet 0.7.7, moment-timezone 2016). Overall, it is a functional but uneven collection; the core connectors are reusable and well-engineered, while many demo scripts are fragile and require manual configuration.
This repository serves as the companion code for the O'Reilly book 'Practical MLOps', providing a collection of examples spanning the MLOps lifecycle. It is a hybrid repository with a dominant data research character, containing Jupyter notebooks for exploratory data analysis (e.g., COVID-19 trends, sugar intake correlations), AutoML experiments (Ludwig text classification), and model interoperability scripts (ONNX conversion). It also includes application-like components such as a Flask-based model serving container, CLI tools for CSV linting, and cloud deployment scripts for AWS, Azure, and GCP. The code is organized by book chapters and demonstrates practical MLOps patterns, though many examples are simplified for educational purposes rather than production readiness. The most reusable asset is the modularized CSV linter, which provides a well-structured CLI for data quality checks.
This repository is a data collection and publishing pipeline for Product Hunt daily launches. It ingests data from the Product Hunt API (via an external consumer app identified by the UTM campaign 'decohack') and produces one markdown file per day, with each file containing structured entries for products: title, tagline, description, keywords, vote counts, comment counts, featured status, and links. The markdown files are the primary data artifact and are consistently formatted, making them parseable for trend analysis, competitive research, or AI training corpora. Supporting Python scripts handle image repair (with exponential backoff for rate limits), WordPress publishing via REST API, and markdown generation. GitHub Actions workflows automate generation, image fixing, and publishing. The repository is best classified as a data pipeline rather than an application: it exposes no API endpoints, UI, or library interface, and its value lies in the accumulated dataset and the automation around it. However, it is a hybrid in that it contains application-like scripts and CI/CD workflows. The data lineage is straightforward: Product Hunt API responses are transformed into daily markdown files, then optionally enriched with images and published to WordPress. The dataset in the analyzed chunk is incomplete (only a subset of dates is present), and there is no schema enforcement, validation, or test suite. The pipeline depends on external services (Product Hunt API, OpenAI, WordPress) that may change or rate-limit, and the hardcoded OpenAI model (gpt-4o-mini) is a drift risk. The markdown format is human-readable but less machine-friendly than JSON/CSV, and no README or data dictionary documents the schema or provenance. Although the last commit is recent, the data files in the analyzed chunk stop at 2025-09-28, suggesting the ingestion pipeline may not be running reliably.
This repository contains the lab materials for the AWS Data Engineering Immersion Day, a guided workshop demonstrating how to build a data lake and analytics pipeline on AWS using services like AWS Database Migration Service (DMS), AWS Glue, Amazon Athena, Amazon QuickSight, Amazon Redshift Spectrum, and AWS Lake Formation. The repo includes step-by-step PDF lab guides, CloudFormation templates for infrastructure provisioning, sample data loading scripts, and a self-contained anomaly detection example using Apache Flink with Random Cut Forest. While the Flink streaming job is a real application, the primary purpose of the repo is instructional, not a production deployment or a self-contained research project. The data lineage involves ingesting ticketing data from a PostgreSQL database via DMS into S3, transforming with Glue into Parquet, and analyzing with Athena/Redshift. Clickstream data is generated into Kinesis streams and processed in real-time with Kinesis Analytics (Flink) to detect anomalies. The repo is hybrid: it has both application code (Flink Java) and data pipeline orchestration, but the dominant character is a workshop with infrastructure-as-code and documentation.