Miller (mlr) is a command-line data processing tool for structured text formats including CSV, TSV, JSON, JSON Lines, YAML, DKVP, and others. It operates as a streaming pipeline: input records are read from files or stdin, processed through a chain of transformer verbs (e.g., cut, sort, join, stats1, put, filter), and written to output in any supported format. The tool includes a domain-specific language (DSL) for record-level transformations, with support for user-defined functions, subroutines, loops, and higher-order functions. Miller is designed for performance, using Go channels for concurrent streaming, batch processing, and a slab allocator for efficient memory use. The architecture is a monolithic CLI application written in Go with no external runtime dependencies. Key entry points include the main binary (cmd/mlr/main.go), the entrypoint dispatcher (pkg/entrypoint), and the terminal subcommands (help, repl, script, regtest, mcp). The codebase is organized into packages for input readers, output writers, transformers (verbs), DSL parsing and execution, CLI flag handling, and a core value type system (Mlrval). Miller also provides an MCP (Model Context Protocol) server for AI agent integration, shell completion scripts, and a REPL for interactive use. Miller is not a research project; it is a production-grade data munging tool comparable to awk, sed, cut, join, and sort but for name-indexed data. Its primary data flows involve reading heterogeneous records, applying transformations via verbs or DSL expressions, and writing results. The tool handles compressed files, in-place editing, and output redirection to files or pipes.
This repository is a curated DNS blocklist project providing domain lists for categories like ads, malware, phishing, tracking, and more. It is primarily a data resource, but includes a sophisticated Python-based build pipeline, automated issue triage, upstream monitoring, and CI/CD workflows. The codebase is config-driven, with reusable modules for domain normalization, validation, format conversion, and merging. The project outputs lists in multiple formats (hosts, AdGuard, dnsmasq, plain domains) and is designed for integration with Pi-hole, AdGuard Home, and similar tools. While the domain lists are the primary artifact, the supporting infrastructure demonstrates strong data engineering practices, including automated testing, structured logging, and exception handling. The repository has a large issue backlog but active maintenance with frequent commits and automated workflows.
This repository is a data pipeline that transforms crowdsourced survey responses about Chinese university quality of life into a structured MkDocs static site. The core ETL script (`questionnaires/main.py`) ingests a CSV of desensitized survey results, applies name normalization via alias mapping, blacklist/whitelist filtering, and splits data into current and archived sets based on a cutoff date. It generates per-university Markdown files with Q&A sections, a navigation structure, and a MkDocs configuration. The pipeline is triggered by GitHub Actions on pushes to master, pushing generated content to a separate branch and to the production website repository. The data is licensed under CC BY-NC-SA 4.0. While the code is functional and well-organized for its purpose, it is tightly coupled to the specific survey schema and data files, limiting direct reuse as a library. The most reusable component is the `FilenameMap` class for generating unique slugs from entity names.
This repository implements FiveThirtyEight's NFL Elo forecasting model and evaluation framework. The data lineage flows from two raw CSV sources: `data/nfl_games.csv` containing historical game results (1920-present) with pre-computed Elo win probabilities, and `data/initial_elos.csv` providing starting Elo ratings for each franchise. The `forecast.py` module implements the core Elo engine: it initializes team ratings, applies season-to-season reversion (retaining 2/3 of prior rating, reverting 1/3 toward 1505), incorporates home-field advantage (65 Elo points), and updates ratings post-game using a margin-of-victory multiplier (log(pd+1) * 2.2/(elo_diff*0.001+2.2)). Special hardcoded reversions handle franchise relocations/expansions. The `util.py` module handles CSV parsing with type coercion and evaluates forecasts using a Brier-score-based points system (25 - 100*brier, doubled for playoffs). The `eval.py` script orchestrates the full pipeline: load games, generate forecasts, evaluate against FiveThirtyEight's baseline. The repo answers the research question: 'Can alternative forecasting models beat FiveThirtyEight's Elo model on historical NFL data?'
The repository is the official code bundle for the book 'Pandas Cookbook - Second Edition' by Packt Publishing. It contains Jupyter notebooks for each chapter, demonstrating pandas functionality for data manipulation, analysis, and visualization. The notebooks use a variety of real-world datasets to illustrate concepts such as data cleaning, filtering, grouping, reshaping, and time series analysis. Additionally, there are two demo directories that provide a reusable module for loading and cleaning Kaggle survey data, along with tests. The repository serves as an educational resource for learning pandas through practical examples.
This repository is a curated collection of Chinese vocabulary lists for standardized tests (HSK 3.0, BCT, YCT), enriched with pinyin, zhuyin, part-of-speech tags, frequency data, and English definitions. It includes Python and JavaScript scripts for data cleaning, format conversion (TXT/TSV/XLSX), POS tag correction using SUBTLEX frequency corpus, and pinyin tone conversion. The data is organized by test level and version (2021 and 2025 HSK standards), with multiple output formats suitable for flashcard apps (Anki, Pleco) and language learning tools. The repository is primarily a data asset with supporting ETL scripts, not a production application. It demonstrates solid data engineering practices for language resource curation, though it suffers from some file duplication and hardcoded paths.