$ show zer0yu/CyberSecurityRSS
This repository is a curated, bilingual (English/Chinese) OPML feed collection focused on cybersecurity and AI, containing approximately 1000+ RSS/Atom feed URLs organized into 12 categories (e.g., Crypto, Pwn, WebSecurity, MobileSecurity, AI). It serves as the data backbone for a personal intelligence pipeline, designed to feed into AI-curated daily security digests. The repo includes automation scripts for feed health monitoring (opml_sync.py with HTTP retries, gzip/deflate decoding, and stateful failure tracking), interactive feed addition (add_feed_to_tiny.py), and optional LLM-assisted pruning (prune_opml.py). A GitHub Actions workflow runs sync operations on push and validates PRs. The actual value lies in the carefully vetted list of high-quality security sources, including many from the Chinese security community, with automated dead-link detection and deduplication. The scripts are well-structured and testable, but the repository's primary deliverable is the curated data, not the application logic.
The LLM identified 3 files worth highlighting for reuse:
A persistent health state file that tracks per-URL consecutive hard failures, transient failures, and last error reason. Enables the sync script to delete feeds only after exceeding a hard-failure threshold, avoiding false positives from transient network issues. This schema is directly reusable for any feed or URL health monitoring system.
Contains `IncrementalContentDecoder` for streaming gzip/deflate decompression with raw-deflate fallback, `HttpFeedChecker` with retry and error classification (hard vs transient), `build_next_health_state` for stateful health tracking, and `sync_tiny_to_full` for generic OPML merge with category fallback. These components are modular, testable, and can be extracted for any feed aggregation or health-checking tool.
Provides `parse_feed_metadata_stream` for memory-bounded incremental RSS/Atom/RDF metadata extraction (early-exit on title+link) and reusable OPML tree helpers (`build_category_map`, `ensure_category`, `iter_rss_nodes`, `resolve_category_name`). These functions are clean, tested, and applicable to any OPML editing or feed metadata extraction task.