$ show StevenBlack/hosts
This repository is a data pipeline that aggregates and consolidates hosts files from 20+ upstream sources (adaway.org, mvps.org, someonewhocares.org, yoyo.org, URLHaus, KADhosts, hostsVN, and others) into unified blocklists for adware, malware, tracking, and optional categories (porn, gambling, fake news, social media). The core pipeline (`updateHostsFile.py`) fetches remote hosts files via HTTP, parses them, deduplicates entries, applies whitelist/blacklist filters, and generates a primary `hosts` file plus 30+ alternate permutations in `alternates/`. Each source has metadata in `update.json` (URL, license, frequency). The `makeHosts.py` script orchestrates full matrix generation of all extension combinations. Output includes stats tracking (`stats.out`, `stats.png`) and auto-generated READMEs via `updateReadme.py`. The repo includes Docker support, GitHub Actions CI (lint, test, multi-Python), CodeQL scanning, and automated dependency updates via Dependabot. Data sources are external URLs; the pipeline is idempotent and designed for scheduled runs.
The LLM identified 5 files worth highlighting for reuse:
Main ETL script that fetches hosts from 20+ configured sources (defined in data/*/update.json), parses lines matching '0.0.0.0 domain' or '127.0.0.1 domain', deduplicates across sources, applies whitelist/blacklist filters, supports optional extensions (porn, gambling, fakenews, social), and writes unified hosts file. Handles HTTP caching, pause flags, and per-source licensing. Highly reusable pattern for multi-source blocklist aggregation.
Orchestrates generation of all 30+ alternate hosts files by recursively combining base sources with every permutation of 4 extensions (fakenews, gambling, porn, social). Calls updateHostsFile.py with appropriate flags for each combination, producing both unified+extensions and extensions-only variants. Clean recursive combinatorial pattern reusable for any multi-dimensional build matrix.
Generates README.md for each alternate hosts file from a single template (readme_template.md) using readmeData.json (entries count, sources metadata). Injects source tables, extension badges, entry counts, and size history graphs. Demonstrates clean separation of content and presentation for data-driven documentation.
Each data source has an update.json declaring name, description, homeurl, fetch URL, license, update frequency, and pause flag. This declarative metadata drives the pipeline (fetch URL, licensing compliance, scheduling hints) and documentation. Pattern reusable for any multi-source data ingestion system requiring provenance tracking.
Bash script that uses `git cat-file --batch` to efficiently extract readmeData.json from all tags, parsing entry counts per tag date to build stats.out time series. Avoids repeated git show calls via streaming blob parsing. Reusable pattern for tracking dataset size evolution in git-tagged data repositories.