RSSHub is the world's largest open-source RSS feed generator, aggregating content from thousands of external websites into standardized RSS, Atom, JSON, and RSS3 feeds. It is a TypeScript application built on the Hono web framework, with a modular architecture where each source (news site, social platform, university notice board, government agency, etc.) is implemented as a self-contained 'route' module that fetches pages or APIs, parses HTML with Cheerio, and optionally renders rich descriptions using JSX. The project supports multiple deployment targets including Node.js, Docker, Vercel, and Cloudflare Workers, and integrates caching (Redis/LRU), middleware for access control, ETag/conditional requests, error tracking (Sentry, Honeybadger), and anti-hotlinking. The codebase is highly organized with consistent route definitions, extensive utility functions (date parsing, HTTP caching, header generation), and a strong emphasis on type safety. While the sheer volume of routes (thousands) introduces maintenance burden and some inconsistency (legacy `got` vs modern `ofetch`), the overall architecture is clean, well-tested for core infrastructure, and exhibits strong engineering practice. This is not a data-research repository; it is a production-grade web service for feed aggregation.
Cobalt is a privacy-focused media downloader web application with a SvelteKit frontend and Node.js/Express API backend. Users paste a URL from supported platforms (YouTube, TikTok, Instagram, Twitter/X, Vimeo, Reddit, SoundCloud, Twitch, Bilibili, and 15+ others) and receive a downloadable media file. The architecture is a monorepo with three packages: @imput/cobalt-web (static frontend), @imput/cobalt-api (processing backend), and @imput/version-info (build-time version helper). The API acts as a fancy proxy - it fetches media streams from source platforms, optionally remuxes/transcodes via FFmpeg (server-side or client-side via libav.js WebAssembly), and streams results back without ever writing to disk. The frontend handles URL input, settings management (localStorage with migration), a processing queue with Web Workers for client-side FFmpeg, i18n (EN/RU), and PWA features. Key data flows: user submits URL → API validates/matches service → extracts stream URLs → returns tunnel/redirect/picker response → frontend either streams directly, proxies via API, or processes locally via Web Workers. External dependencies: FFmpeg (via ffmpeg-static), youtubei.js for YouTube, hls-parser, undici for HTTP, Redis (optional for rate limiting), Cloudflare Turnstile (optional bot protection).
VidBee is a multi-platform media downloader and transcript workspace. The primary product is an Electron desktop app that downloads video/audio from 1000+ sites via yt-dlp, runs local speech-to-text transcription (sherpa-onnx Whisper/SenseVoice/Qwen3/Parakeet families) with speaker diarization in an isolated worker process, and provides an AI agent chat, summarization, translation, and FAQ layer over the resulting transcripts. It also ships a self-hosted Fastify API with a TanStack Start web client, a CLI wrapper around yt-dlp that can run fully in-process or connect to the desktop/API, and a browser extension (WXT) that hands video URLs to the desktop app via a token-authenticated loopback HTTP bridge. An RSS subscription engine with SQLite CAS leader election supports feed auto-download. Architecturally this is a pnpm monorepo built around host-neutral packages: packages/task-queue (FSM-driven job queue with priority scheduler, retry backoff, watchdog, and SQLite/memory persistence adapters), packages/transcription (ASR pipeline, caption parsing, speaker clustering, chunk manifests), packages/downloader-core (yt-dlp args, format selection, cookie handling, oRPC contracts), packages/subscriptions-core (feed parsing, scheduler, leader election), plus packages/ui, packages/i18n, packages/db, and packages/logger. The desktop, API, CLI, and web hosts share these packages, and oRPC contract schemas are explicitly kept byte-equal across hosts. The desktop renderer uses React 19, TanStack Router, Jotai, and Tailwind 4; the main process uses an IPC-decorator pattern with roughly fifteen typed services. Engineering quality is consistently above average: shared contracts with zod validation, swappable persistence adapters, a pure-function task FSM, security-conscious extension token auth, and an SSRF-guarded remote image proxy. The main weaknesses are duplication and sync fragility: near-identical download UI components exist in both desktop and web apps, several overlapping error classifiers must be kept manually in sync, the API hardcodes APP_VERSION separately from package versions, and legacy migration/back-compat paths (task-queue projection to legacy DownloadItem shapes, migrate-history scripts) accumulate complexity. This is an application repository, not data research: the ASR and speaker-clustering code is product engineering rather than a research artifact, though the transcription package is a genuinely reusable pipeline.
This is a mature, actively maintained AI agent skill called 'last30days' that enables agents (Claude Code, Codex, Grok, Gemini CLI, MCP hosts) to research any topic by aggregating content from 20+ social and news platforms (Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, arXiv, Techmeme, Digg, and more) over a rolling 30-day window. The system is primarily a CLI engine (Python) with a companion Go MCP server for Claude Desktop. It employs a sophisticated pipeline: query planning by intent, parallel retrieval from multiple backends (keyless APIs, paid scrapers, CLI tools, LLM-based search), source-specific normalization, engagement-weighted relevance scoring, cross-source clustering, and deterministic fusion (weighted RRF) with per-author caps and diversity floors. Output is rendered as structured JSON, compact text, or self-contained HTML with optional publishing and Atom feeds. The codebase emphasizes security (credential hygiene, no browser cookie reads in unattended mode), determinism (fixture-based evaluation harness with grounding/recency/coherence scores), and graceful degradation (per-source fallback chains, never-raise error envelopes). While classified as an application (it's a skill/plugin for agent hosts), it exhibits strong data engineering traits: source adapters, normalization pipelines, scoring models, and an offline evaluation framework.
MoneyPrinterTurbo is a production short-video generation platform. Given a topic or script, it orchestrates LLM-based script/keyword generation, multi-provider TTS (Edge, Azure, Gemini, ElevenLabs, etc.), video material sourcing (Pexels, Pixabay, Coverr, several AI video generators), subtitle generation (Whisper or TTS timestamps), background music, and final video composition via moviepy/FFmpeg. It exposes three consistent interfaces: a FastAPI REST API, a Streamlit WebUI, and a CLI. All three share the same underlying task pipeline in `app/services/task.py`, which manages state, progress, and cross-posting to YouTube/TikTok/Instagram. The architecture is a service-oriented monolith: the core pipeline is a linear sequence of stages (script → terms → audio → subtitles → materials → video → optional BGM/cross-post). External integrations are abstracted behind provider-specific modules (e.g., `voice.py`, `material.py`, `bgm.py`). A Redis-backed or in-memory state manager tracks task progress. The configuration is runtime-editable and stored in a TOML file. The WebUI is a large Streamlit single-page application with dozens of provider-specific controls, billing confirmations, and i18n support for 12+ languages. The codebase demonstrates strong engineering practices: extensive test coverage (200+ test files), structured error handling with per-provider error classes, atomic file writes, path traversal protection, and security-conscious API key handling. However, it also shows signs of rapid growth: the WebUI is a monolithic 1300+ line script, provider integration logic is occasionally duplicated, and the dependency list is large (many third-party APIs). The repository is actively maintained with recent commits and good issue triage. It is not a research project; there are no datasets, model training, or analytical queries.
This repository provides a Node.js HTTP service that generates TikTok API signatures (X-Bogus and X-Gnarly) required to access TikTok's data endpoints. It uses Puppeteer to run a headless Chromium browser, injects a bundled copy of TikTok's web SDK, and computes signatures by intercepting fetch requests. The service also includes a pure Node.js implementation for X-Gnarly using ChaCha20. It exposes POST /signature and POST /fetch endpoints, manages a request queue, and automatically refreshes the browser session after 500 signatures or 30 minutes. The repository includes example scripts for fetching user videos, comments, trending content, and search results, demonstrating typical TikTok data scraping workflows. While the core functionality is well-implemented, the reliance on static SDK copies and a heavy browser dependency introduces maintenance risks as TikTok updates its anti-scraping measures.