$ show mvanhorn/last30days-skill
This is a mature, actively maintained AI agent skill called 'last30days' that enables agents (Claude Code, Codex, Grok, Gemini CLI, MCP hosts) to research any topic by aggregating content from 20+ social and news platforms (Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, arXiv, Techmeme, Digg, and more) over a rolling 30-day window. The system is primarily a CLI engine (Python) with a companion Go MCP server for Claude Desktop. It employs a sophisticated pipeline: query planning by intent, parallel retrieval from multiple backends (keyless APIs, paid scrapers, CLI tools, LLM-based search), source-specific normalization, engagement-weighted relevance scoring, cross-source clustering, and deterministic fusion (weighted RRF) with per-author caps and diversity floors. Output is rendered as structured JSON, compact text, or self-contained HTML with optional publishing and Atom feeds. The codebase emphasizes security (credential hygiene, no browser cookie reads in unattended mode), determinism (fixture-based evaluation harness with grounding/recency/coherence scores), and graceful degradation (per-source fallback chains, never-raise error envelopes). While classified as an application (it's a skill/plugin for agent hosts), it exhibits strong data engineering traits: source adapters, normalization pipelines, scoring models, and an offline evaluation framework.
The LLM identified 5 files worth highlighting for reuse:
A well-crafted HTTP client layer that provides retry with exponential backoff, token-bucket rate limiting, context-aware failure capture (maps status codes to typed outcomes like RATE_LIMITED, AUTH_FAILED, SCHEMA_DRIFT), and an offline fixture recording/replay system for deterministic testing. Credentials are scrubbed from URLs and response bodies. This module could be extracted as a standalone Python utility for any project needing resilient HTTP calls with instrumentation.
A subprocess runner that kills process groups, escalates SIGTERM to SIGKILL on timeout, suppresses PID-callback exceptions, and guards against Windows AttributeError for killpg. Demonstrates thorough handling of edge cases in subprocess lifecycle. Could be reused in any project that needs reliable external command execution with timeout and cleanup.
Implements weighted Reciprocal Rank Fusion (RRF) with per-source diversity reservation and per-author caps (≤3) to prevent domination by prolific accounts. Includes URL normalization and deduplication, first-party author allowance, and source diversity floor enforcement. This is a reusable ranking fusion algorithm that could be applied to any multi-source retrieval system.
Go embed.FS-based extraction of a Python engine into a versioned cache directory. Uses a .version sentinel file, atomic rename, per-cache-dir sync.Once, and a cache dir override. Demonstrates a clean pattern for packaging a script interpreter alongside a Go binary. Reusable for any MCP server or CLI that needs to embed and execute external scripts.
Near-duplicate detection using n-gram Jaccard similarity and token Jaccard similarity, with prepared text similarity for job postings. Includes URL normalization for deduplication. This is a standalone text similarity utility that could be reused in any content aggregation or deduplication pipeline.