$ show dataabc/weiboSpider
This is a CLI-based web scraper for Sina Weibo (微博) user data and posts. It scrapes user profiles, weibo contents, images, and videos from the mobile version of Weibo (weibo.cn) using a combination of synchronous (requests, lxml) and asynchronous (aiohttp) HTTP requests. The architecture is a pipeline: parsers extract structured data from HTML, downloaders asynchronously fetch media files, and writers persist results to multiple formats (CSV, TXT, JSON, MySQL, MongoDB, SQLite, Kafka, HTTP POST). The spider is driven by a config file (config.json) that controls user id lists, date ranges, filtering, and download options. It supports retry logic, random delays, and global wait periods to avoid rate limiting. External dependencies include requests, lxml, aiohttp, browser_cookie3, and absl-py; optional dependencies add database and message queue support. The codebase is modular with clear separation of concerns, includes unit tests with mocked HTML fixtures, and demonstrates solid engineering practices for a data collection tool.
The LLM identified 2 files worth highlighting for reuse:
An abstract base class for async file downloading with configurable retry logic, connect and read timeouts, and automatic error logging to a 'not_downloaded.txt' file. The `download_one_file` method implements a retry loop with exponential backoff (via random sleep) and aiohttp timeouts, making it resilient to transient network failures. The `download_files` method iterates over a list of objects, calling `handle_download` for each. This class is self-contained and can be extracted and reused in any async scraping project that needs to download files from URLs with retry capabilities.
A collection of utility functions for HTML fetching, text cleaning, and Weibo-specific URL transformations. Notable functions include `handle_html` and `handle_html_async` for fetching and parsing HTML with cookie handling, `handle_garbled` for cleaning Unicode garbled text, `bid2mid` for converting Weibo's base62 IDs to numeric mids, and `to_video_download_url` that extracts the actual video download URL from a Weibo video page by calling the video object API. These functions are isolated (no project-specific imports beyond standard libraries) and could be reused in other Weibo-related scraping tools or generalized for other social media platforms.