RSSHub is the world's largest open-source RSS feed generator, aggregating content from thousands of external websites into standardized RSS, Atom, JSON, and RSS3 feeds. It is a TypeScript application built on the Hono web framework, with a modular architecture where each source (news site, social platform, university notice board, government agency, etc.) is implemented as a self-contained 'route' module that fetches pages or APIs, parses HTML with Cheerio, and optionally renders rich descriptions using JSX. The project supports multiple deployment targets including Node.js, Docker, Vercel, and Cloudflare Workers, and integrates caching (Redis/LRU), middleware for access control, ETag/conditional requests, error tracking (Sentry, Honeybadger), and anti-hotlinking. The codebase is highly organized with consistent route definitions, extensive utility functions (date parsing, HTTP caching, header generation), and a strong emphasis on type safety. While the sheer volume of routes (thousands) introduces maintenance burden and some inconsistency (legacy `got` vs modern `ofetch`), the overall architecture is clean, well-tested for core infrastructure, and exhibits strong engineering practice. This is not a data-research repository; it is a production-grade web service for feed aggregation.
This is a Go-based video parsing tool that removes watermarks from videos across 20+ Chinese social media platforms (Douyin, Kuaishou, Bilibili, Weibo, etc.). It provides three interfaces: a CLI tool (using Cobra subcommands `serve`, `parse`, `id`, `version`), an HTTP API (Gin-based with v1 RESTful endpoints and legacy backward-compatible routes), and a reusable Go library. The architecture is a monolith with a clean separation: `parser/` package contains platform-specific parsers implementing strategy pattern interfaces, `cmd/` package handles CLI and HTTP server logic with a middleware stack (recovery, CORS, rate limiting, basic auth, logging), and `utils/` provides URL extraction. The system is stateless, uses no database, and relies on HTTP scraping and API calls to extract video metadata. Notable engineering craftsmanship includes a custom anti-crawling a_bogus signature generation for Douyin's native detail API (SM3 hash, RC4 encryption), a concurrent-safe proxy injection via atomic.Value, and a well-tested middleware stack with IP-based rate limiting using sync.Map and periodic cleanup. The codebase has comprehensive unit and integration tests, CI/CD pipelines, and Docker support.
PicGo is a mature, cross-platform Electron desktop application for image uploading, primarily targeted at content creators and developers. Its core workflow allows users to upload images via drag-and-drop, clipboard, file picker, or global hotkey, and automatically copies the resulting URL (in Markdown, HTML, or custom format) to the clipboard. The application supports multiple image hosting providers (e.g., GitHub, SM.MS, Qiniu, Alibaba OSS, Tencent COS, Imgur, S3, WebDAV) via a flexible plugin architecture. A built-in HTTP server enables integration with third-party tools. The renderer is built with React, TypeScript, TanStack Router, Zustand for state management, and a comprehensive UI component library (shadcn/ui style). The main process handles IPC, window management, system tray, global shortcuts, file operations, and a plugin lifecycle. PicGo Cloud integration provides user accounts, configuration sync (with optional E2E encryption), cloud album storage, and usage analytics. The codebase includes extensive internationalization (5 languages), a robust test suite (Vitest), and automated CI/CD for multi-platform builds. While the project has some technical debt (incomplete Vue migration, numerous TODO comments, patched dependencies), it demonstrates strong engineering practices, modular architecture, and a well-documented plugin ecosystem.
This is a CLI-based web scraper for Sina Weibo (微博) user data and posts. It scrapes user profiles, weibo contents, images, and videos from the mobile version of Weibo (weibo.cn) using a combination of synchronous (requests, lxml) and asynchronous (aiohttp) HTTP requests. The architecture is a pipeline: parsers extract structured data from HTML, downloaders asynchronously fetch media files, and writers persist results to multiple formats (CSV, TXT, JSON, MySQL, MongoDB, SQLite, Kafka, HTTP POST). The spider is driven by a config file (config.json) that controls user id lists, date ranges, filtering, and download options. It supports retry logic, random delays, and global wait periods to avoid rate limiting. External dependencies include requests, lxml, aiohttp, browser_cookie3, and absl-py; optional dependencies add database and message queue support. The codebase is modular with clear separation of concerns, includes unit tests with mocked HTML fixtures, and demonstrates solid engineering practices for a data collection tool.
A monolithic Python application for crawling public data from the Chinese social network Weibo (Sina Weibo). It retrieves user profiles, weibo posts (including text, images, videos, live photos), comments, and reposts via the Weibo mobile API. Data can be exported to CSV, JSON, SQLite, MySQL, MongoDB, Markdown, or forwarded to a custom HTTP endpoint. It provides a CLI entry point for scheduled crawling (using the `schedule` library) and a Flask-based REST API for remote trigger and data retrieval. The codebase is heavily centralized—the core `Weibo` class (~3000 lines) handles fetching, parsing, downloading media, writing to multiple outputs, and anti‑ban throttling. A lightweight LLM analysis module can optionally enrich each weibo with sentiment, summary, and anomaly detection. The repository is community‑maintained with 4.6k stars, 402 open issues, and frequent commits as of mid‑2026. It is primarily an application but has a strong data‑collection character; the data outputs are suitable for subsequent research or archiving.