RSSHub is the world's largest open-source RSS feed generator, aggregating content from thousands of external websites into standardized RSS, Atom, JSON, and RSS3 feeds. It is a TypeScript application built on the Hono web framework, with a modular architecture where each source (news site, social platform, university notice board, government agency, etc.) is implemented as a self-contained 'route' module that fetches pages or APIs, parses HTML with Cheerio, and optionally renders rich descriptions using JSX. The project supports multiple deployment targets including Node.js, Docker, Vercel, and Cloudflare Workers, and integrates caching (Redis/LRU), middleware for access control, ETag/conditional requests, error tracking (Sentry, Honeybadger), and anti-hotlinking. The codebase is highly organized with consistent route definitions, extensive utility functions (date parsing, HTTP caching, header generation), and a strong emphasis on type safety. While the sheer volume of routes (thousands) introduces maintenance burden and some inconsistency (legacy `got` vs modern `ofetch`), the overall architecture is clean, well-tested for core infrastructure, and exhibits strong engineering practice. This is not a data-research repository; it is a production-grade web service for feed aggregation.
Cobalt is a privacy-focused media downloader web application with a SvelteKit frontend and Node.js/Express API backend. Users paste a URL from supported platforms (YouTube, TikTok, Instagram, Twitter/X, Vimeo, Reddit, SoundCloud, Twitch, Bilibili, and 15+ others) and receive a downloadable media file. The architecture is a monorepo with three packages: @imput/cobalt-web (static frontend), @imput/cobalt-api (processing backend), and @imput/version-info (build-time version helper). The API acts as a fancy proxy - it fetches media streams from source platforms, optionally remuxes/transcodes via FFmpeg (server-side or client-side via libav.js WebAssembly), and streams results back without ever writing to disk. The frontend handles URL input, settings management (localStorage with migration), a processing queue with Web Workers for client-side FFmpeg, i18n (EN/RU), and PWA features. Key data flows: user submits URL → API validates/matches service → extracts stream URLs → returns tunnel/redirect/picker response → frontend either streams directly, proxies via API, or processes locally via Web Workers. External dependencies: FFmpeg (via ffmpeg-static), youtubei.js for YouTube, hls-parser, undici for HTTP, Redis (optional for rate limiting), Cloudflare Turnstile (optional bot protection).
MediaGo is a cross-platform media download manager that enables users to download videos from a wide range of sources, including HLS/m3u8 streams, YouTube, Bilibili, TikTok, Douyin, Xiaohongshu, and direct URLs. It is built as a monorepo with a Go backend (REST API + SSE events), an Electron desktop shell, a web UI (React 19), a mobile player UI, and a Chrome MV3 browser extension. The backend manages a download task queue that orchestrates external binaries (yt-dlp, BBDown, ffmpeg, aria2c, N_m3u8DL-RE) via PTY/exec runners, provides real-time progress through SSE, and includes an MCP server for AI agent integration. The Electron app embeds a Chromium webview for media source sniffing and supports share intents, ad blocking, and device mode. The browser extension detects media resources on pages and dispatches them to the local or remote MediaGo server via HTTP or a custom protocol.
TubeArchivist is a self-hosted YouTube media server that downloads, indexes, and serves YouTube video content. In production, it runs as a Docker-based stack with a Django REST API backend, React/TypeScript frontend, Celery task queue, Redis broker, and Elasticsearch for metadata storage and full-text search. The system subscribes to YouTube channels and playlists via yt-dlp, extracts comprehensive metadata (including subtitles, comments, SponsorBlock segments, and thumbnails), and provides a web UI for browsing, searching, streaming video, and managing downloads. It supports multiple authentication modes (single user, LDAP, forward proxy), scheduled tasks for rescanning subscriptions and backups, and integrates with Apprise for notifications and a membership/SponsorBlock API for ad-skipping. The architecture follows a modular Django app pattern (video, channel, playlist, download, task, appsettings, user, stats) with consistent use of base views, serializers, and search processors. The frontend uses Zustand for state management and a custom API client with CSRF handling. The codebase is actively maintained with comprehensive test coverage and CI/CD pipelines.
VidBee is a multi-platform media downloader and transcript workspace. The primary product is an Electron desktop app that downloads video/audio from 1000+ sites via yt-dlp, runs local speech-to-text transcription (sherpa-onnx Whisper/SenseVoice/Qwen3/Parakeet families) with speaker diarization in an isolated worker process, and provides an AI agent chat, summarization, translation, and FAQ layer over the resulting transcripts. It also ships a self-hosted Fastify API with a TanStack Start web client, a CLI wrapper around yt-dlp that can run fully in-process or connect to the desktop/API, and a browser extension (WXT) that hands video URLs to the desktop app via a token-authenticated loopback HTTP bridge. An RSS subscription engine with SQLite CAS leader election supports feed auto-download. Architecturally this is a pnpm monorepo built around host-neutral packages: packages/task-queue (FSM-driven job queue with priority scheduler, retry backoff, watchdog, and SQLite/memory persistence adapters), packages/transcription (ASR pipeline, caption parsing, speaker clustering, chunk manifests), packages/downloader-core (yt-dlp args, format selection, cookie handling, oRPC contracts), packages/subscriptions-core (feed parsing, scheduler, leader election), plus packages/ui, packages/i18n, packages/db, and packages/logger. The desktop, API, CLI, and web hosts share these packages, and oRPC contract schemas are explicitly kept byte-equal across hosts. The desktop renderer uses React 19, TanStack Router, Jotai, and Tailwind 4; the main process uses an IPC-decorator pattern with roughly fifteen typed services. Engineering quality is consistently above average: shared contracts with zod validation, swappable persistence adapters, a pure-function task FSM, security-conscious extension token auth, and an SSRF-guarded remote image proxy. The main weaknesses are duplication and sync fragility: near-identical download UI components exist in both desktop and web apps, several overlapping error classifiers must be kept manually in sync, the API hardcodes APP_VERSION separately from package versions, and legacy migration/back-compat paths (task-queue projection to legacy DownloadItem shapes, migrate-history scripts) accumulate complexity. This is an application repository, not data research: the ASR and speaker-clustering code is product engineering rather than a research artifact, though the transcription package is a genuinely reusable pipeline.
Podsync is a Go-based service that converts video/audio content from YouTube, Vimeo, SoundCloud, and Twitch into podcast RSS feeds. It downloads media via yt-dlp, stores metadata in BadgerDB, and serves feeds via an HTTP server. The architecture is modular: platform-specific builders (pkg/builder/) query APIs to discover episodes, a downloader (pkg/ytdl/) handles media retrieval, and a feed generator (pkg/feed/) produces iTunes-compatible RSS and OPML. The system supports cron-based scheduling, episode filtering by regex/duration/age, cleanup policies, API key rotation, and hooks for custom workflows. It runs as a standalone binary or Docker container with local or S3 storage.
Pake is a CLI tool that packages any webpage or local web build into a lightweight native desktop application (~5MB) using Tauri (Rust) and the system WebView. It supports macOS, Windows, and Linux, producing installers in dmg/app, msi, deb/AppImage/rpm/zst formats. The architecture is a dual-language system: a TypeScript CLI (Commander.js) orchestrates the build process, generating Tauri configuration files, handling icon conversion, and managing platform-specific builders (MacBuilder, WinBuilder, LinuxBuilder) via a base builder pattern. The Rust Tauri runtime provides the desktop shell with webview injection scripts for features like find-in-page, fullscreen polyfill, style injection, clipboard bridging, and auth/popup routing. The project includes extensive unit and integration tests, a GitHub Action for CI/CD, and machine-readable JSON output for agent integration. It is not a data or research project; it is a pure application packaging tool.
This is a mature, actively maintained AI agent skill called 'last30days' that enables agents (Claude Code, Codex, Grok, Gemini CLI, MCP hosts) to research any topic by aggregating content from 20+ social and news platforms (Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, arXiv, Techmeme, Digg, and more) over a rolling 30-day window. The system is primarily a CLI engine (Python) with a companion Go MCP server for Claude Desktop. It employs a sophisticated pipeline: query planning by intent, parallel retrieval from multiple backends (keyless APIs, paid scrapers, CLI tools, LLM-based search), source-specific normalization, engagement-weighted relevance scoring, cross-source clustering, and deterministic fusion (weighted RRF) with per-author caps and diversity floors. Output is rendered as structured JSON, compact text, or self-contained HTML with optional publishing and Atom feeds. The codebase emphasizes security (credential hygiene, no browser cookie reads in unattended mode), determinism (fixture-based evaluation harness with grounding/recency/coherence scores), and graceful degradation (per-source fallback chains, never-raise error envelopes). While classified as an application (it's a skill/plugin for agent hosts), it exhibits strong data engineering traits: source adapters, normalization pipelines, scoring models, and an offline evaluation framework.
Spotube is a cross-platform, open-source Spotify client built with Flutter. It provides a full-featured music player experience without requiring a Spotify Premium subscription by leveraging plugin-based metadata providers and audio sources (YouTube, Piped, Invidious, JioSaavn, etc.). The application supports local playback, playlist management, lyrics (synced and plain), downloads, scrobbling (Last.fm), remote device control via WebSocket, and extensive customization (themes, layouts, audio quality). It uses Riverpod for state management, auto_route for navigation, drift for local SQLite storage, and media_kit for audio playback. The architecture is modular with a plugin system for metadata and audio sources, making it extensible. The codebase is well-structured with reusable patterns like pagination mixins, universal image loading, and type converters. It is actively maintained with nightly builds and a large community.
Open Video Downloader (formerly youtube-dl-gui) is a cross-platform desktop GUI for downloading videos and audio from YouTube and over 1000 other websites, using yt-dlp as its core engine. It provides a queue-based download workflow with features including format/codec selection, subtitle management, SponsorBlock integration, authentication (cookies, tokens, passwords), input filtering (date, size, match filters), and extensive output customization (containers, filename templates, directory organization). The application also manages automatic updates of both itself and its helper binaries (yt-dlp, ffmpeg, ffprobe, deno, AtomicParsley) via a signed manifest. The architecture is a Tauri 2 application with a Rust backend and Vue 3 frontend. The Rust backend handles yt-dlp process management, output parsing (progress, errors, metadata), binary updates, configuration persistence, and concurrent download scheduling. The frontend uses Pinia stores for state management, Vue Router for navigation, and composables for reusable logic. Communication between frontend and backend occurs through Tauri IPC commands and events, with a well-defined command structure in Rust and corresponding listener plugins in TypeScript. Key entry points include src/main.ts (Vue app bootstrap), src-tauri/src/main.rs (Rust entry), src/router.ts (route definitions), and src/plugins/tauriListeners.ts (event listener registration). The primary data flow begins when a user adds a URL: the frontend sends an IPC command to the backend, which runs yt-dlp to fetch metadata, parses the JSON output into structured models, and returns the data to the frontend. The user then configures download options, and the backend spawns a yt-dlp download process, streaming progress events back to the frontend for real-time UI updates. External dependencies include yt-dlp, ffmpeg, ffprobe, deno, AtomicParsley, the SponsorBlock API, system keyring (via Tauri stronghold), and Sentry for error reporting.
YouDub WebUI is a full-stack video localization platform that enables users to import videos, transcribe speech via Whisper, translate via OpenAI-compatible APIs, clone voices with VoxCPM2, separate audio with Demucs, align subtitles with Qwen3 Forced Aligner, and export dubbed videos or subtitles. The architecture is plugin-based (Cordis framework) with a Node.js/TypeScript host orchestrating Python subprocesses for ML inference, a Next.js web frontend, and native desktop apps for macOS (Swift) and Windows (C#). The system features a task engine with queuing, cancellation, retry, and partial rerun; a worker protocol using NDJSON over stdin/stdout; and a comprehensive setup service that downloads and verifies model weights. While primarily an application, it includes research-grade model consumption and detailed validation records documenting real-media runs with SHA-256 hashes and human listening feedback. The codebase is actively maintained with recent commits, CI/CD pipelines, and extensive unit tests for contracts, protocols, and permissions.
Lux is a fast and simple video downloader CLI tool written in Go. It supports downloading videos, images, and audio from over 40 websites including YouTube, Bilibili, TikTok, Instagram, Twitter, and many Chinese streaming platforms. The tool operates as a single binary with a rich set of command-line options for selecting quality, downloading playlists, resuming interrupted downloads, multi-threaded downloading, and integration with aria2 for RPC-based downloads. Architecturally, Lux follows a plugin-based extractor pattern. Each supported website has its own extractor module under `extractors/`, registered via `init()` functions into a global map keyed by domain. The main entry point is `main.go`, which parses CLI flags via `urfave/cli` and dispatches to the appropriate extractor. Extracted video metadata (streams, quality, size) is passed to the downloader, which handles HTTP requests, progress bars, file merging via ffmpeg, and optional subtitle embedding. Key data flows: URL input → domain detection → extractor selection → HTML/API scraping → stream metadata extraction → downloader (single or multi-threaded) → file output. External dependencies include ffmpeg for merging and subtitle embedding, and various third-party Go libraries for HTTP requests, HTML parsing, JavaScript execution (for TikTok signing), and progress bars. The codebase is well-organized with clear separation of concerns, extensive test coverage for individual extractors, and CI workflows that run weekly to verify extractor functionality. While many extractors rely on fragile regex-based scraping of HTML pages, the overall architecture is solid and the downloader implements robust features like resume support and configurable retry logic.
Sockseek is a command-line music downloader for the Soulseek peer-to-peer network, written in C# (.NET). It searches the Soulseek network for songs and albums, ranks candidates using a sophisticated multi-level sorting algorithm (bitrate, format, user reputation, etc.), and downloads the best match. The application supports multiple input sources: free-text search strings, Spotify/YouTube/Bandcamp/MusicBrainz URLs, CSV files, Soulseek links, and list files. It can run as a one-shot CLI tool or as a persistent daemon with an HTTP API and SignalR real-time event hub for remote control and live progress monitoring. Architecturally, the codebase is modular with four main projects: `Sockseek.Core` (core domain logic, search, download orchestration, extractors, file management), `Sockseek.Cli` (CLI entry point, interactive mode, progress rendering), `Sockseek.Server` (ASP.NET Core daemon with SignalR, event coalescing, state store), and `Sockseek.Api` (DTOs, JSON serialization context, client library). The system uses an event-driven architecture for decoupled progress reporting and state management. Extensive unit and integration tests (over 50 test files) and performance benchmarks (BenchmarkDotNet) ensure reliability and performance. Key features include incremental search result projection (album folders, aggregate tracks), a flexible profile system with condition-based auto-profiles, file condition evaluation (bitrate, sample rate, strict title/artist/album matching), fallback downloads via yt-dlp, stale download detection, and post-download command execution with variable substitution. The daemon exposes a full REST API and OpenAPI specification, enabling remote CLI clients and potential third-party integrations.
The Social Media Hacker List repository primarily maintains a curated markdown list of tools and applications for enhancing social media experiences across various platforms (Facebook, Instagram, Twitter, etc.). The list is organized by platform and includes entries such as alternative clients, analytics tools, bots, and content creation utilities. Each entry is manually curated and must meet specific inclusion criteria, such as being created by unaffiliated developers and having recent activity. To ensure the list remains accurate and free of broken or inactive links, the repository includes a Rust-based URL checker application. This tool is executed weekly via a GitHub Actions workflow (check.yaml) to validate every URL in the README.md. For standard URLs, it performs HEAD and GET requests, treating certain HTTP status codes as success (200, 403, 406) with special handling for known edge cases (e.g., 502 for reddit.com, 429 for apps.apple.com). For GitHub repository URLs, it uses the GitHub API to check if the repository is archived or has not been pushed to in the last 365 days, flagging such entries as invalid. The checker application is designed as a standalone, concurrent command-line tool. It reads input from stdin, extracts URLs using a regular expression, and processes them asynchronously with the tokio runtime. The application is containerized via a multi-stage Dockerfile and published to GitHub Container Registry through another workflow (publish.yaml), allowing it to be reused in other contexts. The code demonstrates solid Rust practices, including proper error handling, asynchronous I/O, and modular function design, making its core URL and GitHub checking logic potentially reusable in other projects with minimal adaptation.
NewPipe Extractor is a Java library that provides a unified abstraction for extracting structured metadata (streams, channels, playlists, comments, search results) from multiple streaming platforms including YouTube, SoundCloud, Bandcamp, PeerTube, and media.ccc.de. It reverse-engineers private APIs and scrapes HTML/JSON to produce normalized data models. The library is designed as a headless component for the NewPipe Android app but can be used independently. Its architecture features a service registry, abstract extractor classes, link handler factories for URL normalization, and a collector pattern for batch extraction with error accumulation. The codebase is well-tested with JUnit 5 and a mock/recording downloader system, but YouTube-specific extraction is fragile due to reliance on hardcoded client versions, JavaScript deobfuscation, and frequently changing InnerTube API schemas.
YTDLnis is a feature-rich Android application that serves as a graphical frontend for yt-dlp, enabling users to download video and audio from over 1000 websites. The app follows the MVVM architecture with Room database for local persistence, WorkManager for background download tasks, and a repository pattern for data access. It provides a comprehensive download management system including queue management, scheduling, format selection, metadata embedding, chapter splitting, SponsorBlock integration, and support for custom yt-dlp commands. The application also includes a built-in terminal emulator for advanced users, cookie management, YouTube PoToken generation, and an observe-sources feature for monitoring channels or playlists for new content. The codebase is well-organized into UI fragments, viewmodels, repositories, and utility classes, with extensive use of Kotlin coroutines and Android Jetpack components. While the app is actively maintained and demonstrates solid engineering practices, some modules like UiUtil.kt are monolithic and tightly coupled, and the heavy reliance on external tools (yt-dlp, FFmpeg, Python) introduces ongoing maintenance risks.
The Video Autopilot Kit is a Python-based framework for automating the creation of vertical shorts (YouTube Shorts, Reels, TikTok) and long-form videos. It provides three isomorphic production lines (teaching long-form, vertical shorts, interview show) each built as a knowledge layer → mechanical gate → one-command driver. The system centers around an Editkin v4 durable workflow contract that serves as the single source of truth for planning, evidence binding, audit, atomic apply, and render. Key features include an image-first material system requiring verified source media, a unified filter library of 38 reusable presets, and privacy-first design that avoids storing personal data in the repository. The architecture is modular, with specialized modules for asset management, color grading, audio processing, motion assets, art direction, scene auditing, and quality assurance. Entry points include CLI-driven pipelines for shorts (`src/shorts_autopilot.py`), interviews (`src/interview_autopilot.py`), long-form (`src/longform_maker/`), publishing (`src/publish_hub_cli.py`), and competitor teardown (`src/teardown.py`). The system relies on external tools like `ffmpeg`/`ffprobe` for video processing and optional OCR libraries for competitor analysis, with cross-platform compatibility handled via `src/platform_compat.py`. Despite its strengths, the kit faces maintenance challenges due to hardcoded assumptions about directory structures and platform specifications, as well as dependency drift risks from external tools like ffmpeg and OCR libraries. However, the inclusion of a release manager and self-updater helps mitigate some of these risks by providing deterministic builds and update mechanisms.
SimpMusic is a production-grade Kotlin Multiplatform music player for Android and desktop (JVM) that streams from YouTube Music's undocumented backend. The codebase is organized around a shared Compose Multiplatform UI module (composeApp), Android entry points (androidApp), a desktop entry point (desktopApp), and optional feature modules for Google Cast, Last.fm scrobbling, and crash reporting. A separately referenced core submodule supplies domain and media-service logic. Internally it uses MVVM with Koin dependency injection, Repository interfaces backed by Room entities and DataStore, StateFlow-based ViewModels, and a large custom Compose component library covering home feed, library management, playlist sync, fullscreen player, lyrics, equalizer, podcasts, Android home-screen widgets, and a desktop mini-player. The app is shipped as a packaged consumer application with Play Store metadata, GitHub Actions, fastlane changelogs, and build variants that swap real integrations for empty stubs in FOSS builds.
This repository is a full-stack self-hosted media download manager and library preparation layer centered on yt-dlp. The backend is implemented in Python using aiohttp, SQLite (via SQLAlchemy), and includes a structured feature-based architecture (auth, presets, tasks, conditions, downloads, streaming, notifications, ytdlp integration). The frontend is a Nuxt 3 / Vue 3 single-page application with comprehensive i18n support. Core capabilities include automated download scheduling with cron-like task definitions, conditional filtering, Cloudflare challenge solving, real-time WebSocket updates, streaming via on-the-fly HLS transcoding with hardware acceleration fallback, and a browser extension for quick URL submission. The codebase demonstrates good engineering practices: event bus for decoupled communication, singleton-based service containers, repository pattern for data access, custom migration framework, and extensive async test coverage. Notable external dependencies are yt-dlp (core downloader), ffmpeg (transcoding), curl_cffi (impersonation), and apprise (notifications). While the architecture is modular and well-organized, there is some code duplication across CRUD repositories and migration files, and tight coupling to yt-dlp internals via custom patches. The project is actively maintained with frequent commits, a large test suite, and clear containerization support.
ytDownloader is a cross-platform desktop application built with Electron that provides a graphical interface for downloading videos and audio from YouTube and hundreds of other sites using yt-dlp as the backend download engine. It features a comprehensive download manager supporting single downloads, playlists, and YouTube search, with options for format selection, quality, subtitles, compression, and output templates. The application manages its own yt-dlp and ffmpeg binaries, supports auto-updates, and includes a download history with JSON/CSV export, theming, internationalization (23+ languages), system tray integration, and a global hotkey for quick URL pasting. Architecture: The application follows a standard Electron architecture with a main process (main.js) handling window management, IPC handlers, auto-updater, system tray, and global hotkey. A preload script (preload.js) sets up a context bridge (electronAPI) that exposes filesystem operations, process spawning, clipboard access, and other OS services to the renderer. The renderer process consists of several JavaScript modules (src/index.js, src/playlist.js, src/compressor.js, src/preferences.js, src/history_ui.js, src/common.js) that handle specific views. Communication between renderer and main process occurs via Electron's IPC mechanism (ipcRenderer/ipcMain). User preferences are stored in localStorage and a JSON config file, while download history is persisted as JSON. Key entry points: main.js (Electron main process), preload.js (context bridge), html/index.html (single-page UI shell), src/renderer.js (main renderer logic with YtDownloaderApp class), src/index.js (homepage downloader), src/playlist.js (playlist downloader), src/compressor.js (video compressor), src/preferences.js (settings), src/history_ui.js (history view). External dependencies include yt-dlp (download engine), ffmpeg/ffprobe (media processing), Electron (framework), electron-builder (packaging), electron-updater (auto-updates), systeminformation (GPU detection), and SlimSelect (custom dropdowns). The application bundles these binaries or uses system-installed versions, with fallback mechanisms.