$ show zcyeee/EastMoney_Crawler
This repository implements a web scraper for the EastMoney stock forum (guba.eastmoney.com) using Selenium and MongoDB. It provides two main crawlers: PostCrawler scrapes post listings (title, views, comment count, URL, date, author) from paginated forum pages, and CommentCrawler scrapes individual post pages for comments (content, likes, date, sub-comments). The scraper uses a stealth.js script to evade bot detection, handles errors by restarting the WebDriver, and supports multi-threaded crawling for multiple stock symbols. Data is stored in MongoDB collections named post_<symbol> and comment_<symbol>. The code is a monolithic script with classes for crawling, parsing, and database operations. It is a functional but unremarkable scraper suitable for small-scale data collection, with basic error handling and no testing or dependency management.
The LLM identified 1 files worth highlighting for reuse:
A thin wrapper around pymongo providing basic CRUD operations (insert_one, insert_many, find, find_one, update_one, drop, count_documents) and convenience methods (find_first, find_last). While simple, it encapsulates MongoDB connection details and could be reused in other projects with minimal modification. It demonstrates a clean interface for database interactions, though it lacks advanced features like connection pooling or error handling beyond BulkWriteError.