MediaCrawler is a multi-platform social media data collection tool targeting Chinese platforms: Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu. It uses Playwright browser automation to log in (QR code, phone, cookie) and then scrapes posts, comments, and media files based on search keywords, specific post IDs, or creator IDs. The architecture is a monolith with modular platform-specific crawlers, a shared abstract base layer, and a FastAPI backend that provides a WebUI for visual control. Data can be stored in multiple formats (CSV, JSON, JSONL, SQLite, MySQL, PostgreSQL, MongoDB, Excel). Additional features include a robust media downloader with DASH support, proxy rotation, word cloud generation, and privacy-focused anonymization. The codebase demonstrates solid engineering with design patterns (factory, abstract base, mixin), good test coverage, and a clean separation of concerns, though there is some duplication across platform implementations.
BettaFish (åūŪč) is a multi-agent public opinion analysis system that automates the end-to-end process of gathering, analyzing, and reporting on public sentiment across 30+ Chinese social media platforms. The system is built around four specialized agents: QueryEngine (news breadth search via Tavily), MediaEngine (multimodal search via Bocha), InsightEngine (database mining via SQLAlchemy), and ReportEngine (template-based report generation with intermediate representation and HTML/PDF/Markdown rendering). A ForumEngine coordinates agent collaboration through log monitoring and LLM-generated moderator comments. The architecture is a monolithic Flask application that manages multiple Streamlit sub-applications and CLI tools, with a separate MindSpider crawler subsystem for data collection. While primarily an application, it includes significant data/research components such as sentiment analysis model training scripts (BERT, GPT-2, Qwen3, traditional ML), a Weibo sentiment dataset, and data pipeline scripts. The system relies heavily on external LLM APIs (OpenAI-compatible), search APIs (Tavily, Bocha, Anspire), and system dependencies (Playwright, WeasyPrint). The codebase demonstrates solid engineering with reusable patterns like robust JSON parsing, chart validation/repair, and a hardened GitHub Actions star-history script, but suffers from significant code duplication across engines and tight coupling to external APIs.