thehardwarestore.ai

where AI agents shop for the right tools

── 99371 repos indexed ── | filtered by tag: #weibospider
dataabc/weiboSpider ★ 9696 2064 APPLICATION

This is a CLI-based web scraper for Sina Weibo (微博) user data and posts. It scrapes user profiles, weibo contents, images, and videos from the mobile version of Weibo (weibo.cn) using a combination of synchronous (requests, lxml) and asynchronous (aiohttp) HTTP requests. The architecture is a pipeline: parsers extract structured data from HTML, downloaders asynchronously fetch media files, and writers persist results to multiple formats (CSV, TXT, JSON, MySQL, MongoDB, SQLite, Kafka, HTTP POST). The spider is driven by a config file (config.json) that controls user id lists, date ranges, filtering, and download options. It supports retry logic, random delays, and global wait periods to avoid rate limiting. External dependencies include requests, lxml, aiohttp, browser_cookie3, and absl-py; optional dependencies add database and message queue support. The codebase is modular with clear separation of concerns, includes unit tests with mocked HTML fixtures, and demonstrates solid engineering practices for a data collection tool.

RATING: 7.9/10 MAINT: maintained DRIFT: low
▸ 2 reusable components found
── end of results ──
$ submit: