$ show dataabc/weibo-crawler
A monolithic Python application for crawling public data from the Chinese social network Weibo (Sina Weibo). It retrieves user profiles, weibo posts (including text, images, videos, live photos), comments, and reposts via the Weibo mobile API. Data can be exported to CSV, JSON, SQLite, MySQL, MongoDB, Markdown, or forwarded to a custom HTTP endpoint. It provides a CLI entry point for scheduled crawling (using the `schedule` library) and a Flask-based REST API for remote trigger and data retrieval. The codebase is heavily centralized—the core `Weibo` class (~3000 lines) handles fetching, parsing, downloading media, writing to multiple outputs, and anti‑ban throttling. A lightweight LLM analysis module can optionally enrich each weibo with sentiment, summary, and anomaly detection. The repository is community‑maintained with 4.6k stars, 402 open issues, and frequent commits as of mid‑2026. It is primarily an application but has a strong data‑collection character; the data outputs are suitable for subsequent research or archiving.
The LLM identified 2 files worth highlighting for reuse:
An isolated class that abstracts calls to an OpenAI‑compatible LLM API for sentiment analysis, summary generation, and anomaly detection. It is configurable via `llm_config` in the main config, handles errors gracefully, and returns structured results. Could be extracted and reused in any project requiring text analysis with minimal modification.
A simple, self‑contained module providing two functions: `insert_or_update_user` and `update_last_weibo_id`. These handle CSV file creation, appending rows, and updating a specific field (the last crawled weibo ID). The pattern is generic enough to be reused in any project needing a lightweight key‑value record in CSV format.