This is a CLI-based web scraper for Sina Weibo (微博) user data and posts. It scrapes user profiles, weibo contents, images, and videos from the mobile version of Weibo (weibo.cn) using a combination of synchronous (requests, lxml) and asynchronous (aiohttp) HTTP requests. The architecture is a pipeline: parsers extract structured data from HTML, downloaders asynchronously fetch media files, and writers persist results to multiple formats (CSV, TXT, JSON, MySQL, MongoDB, SQLite, Kafka, HTTP POST). The spider is driven by a config file (config.json) that controls user id lists, date ranges, filtering, and download options. It supports retry logic, random delays, and global wait periods to avoid rate limiting. External dependencies include requests, lxml, aiohttp, browser_cookie3, and absl-py; optional dependencies add database and message queue support. The codebase is modular with clear separation of concerns, includes unit tests with mocked HTML fixtures, and demonstrates solid engineering practices for a data collection tool.
A monolithic Python application for crawling public data from the Chinese social network Weibo (Sina Weibo). It retrieves user profiles, weibo posts (including text, images, videos, live photos), comments, and reposts via the Weibo mobile API. Data can be exported to CSV, JSON, SQLite, MySQL, MongoDB, Markdown, or forwarded to a custom HTTP endpoint. It provides a CLI entry point for scheduled crawling (using the `schedule` library) and a Flask-based REST API for remote trigger and data retrieval. The codebase is heavily centralized—the core `Weibo` class (~3000 lines) handles fetching, parsing, downloading media, writing to multiple outputs, and anti‑ban throttling. A lightweight LLM analysis module can optionally enrich each weibo with sentiment, summary, and anomaly detection. The repository is community‑maintained with 4.6k stars, 402 open issues, and frequent commits as of mid‑2026. It is primarily an application but has a strong data‑collection character; the data outputs are suitable for subsequent research or archiving.