$ show kimwang1978/collect-txt
This repository is an automated IPTV playlist aggregation pipeline. It operates as a daily GitHub Actions workflow that fetches live TV channel URLs from dozens of remote sources, parses them (M3U and plain text), classifies by channel category using hand-maintained dictionary files (e.g., CCTV, provincial channels, sports, movies), applies name normalization via OpenCC, deduplicates, sorts, and outputs merged TXT and M3U files. It also includes a stream validation subsystem (blacklist1.py, blacklist2.py) that checks URL health, records failures with timestamps, and maintains a persistent failure cache and host reputation tracker via a JavaScript state module. Historical snapshots and output archives are stored for rollback. The system is heavily customized for Chinese IPTV sources and lacks automated tests, but the core pipeline pattern and the stream validation state management are reusable.
The LLM identified 2 files worth highlighting for reuse:
A self-contained JavaScript module that manages a persistent JSON-based failure cache for URL health checks. It normalizes URLs to host keys, tracks failure timestamps and counts, implements exponential retry delays based on accumulated failures (e.g., 7+ failures trigger 72-hour cooldown), and maintains host reputation scores. The pattern separates state logic from checking logic and can be extracted for any HTTP health-check system that needs resilient retry scheduling.
A multi-threaded IPTV URL validator using ThreadPoolExecutor and httpx. It reads a list of candidate URLs from a text file, performs HEAD/GET requests to check status, separates working and failed URLs into auto-generated whitelist/blacklist files. The concurrency pattern and output recording are cleanly separated from project-specific logic; could be reused for any bulk URL validation task by replacing the input source and output format.