$ show NanmiCoder/MediaCrawler
MediaCrawler is a multi-platform social media data collection tool targeting Chinese platforms: Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu. It uses Playwright browser automation to log in (QR code, phone, cookie) and then scrapes posts, comments, and media files based on search keywords, specific post IDs, or creator IDs. The architecture is a monolith with modular platform-specific crawlers, a shared abstract base layer, and a FastAPI backend that provides a WebUI for visual control. Data can be stored in multiple formats (CSV, JSON, JSONL, SQLite, MySQL, PostgreSQL, MongoDB, Excel). Additional features include a robust media downloader with DASH support, proxy rotation, word cloud generation, and privacy-focused anonymization. The codebase demonstrates solid engineering with design patterns (factory, abstract base, mixin), good test coverage, and a clean separation of concerns, though there is some duplication across platform implementations.
The LLM identified 3 files worth highlighting for reuse:
A generic, platform-agnostic media downloader that supports retry with exponential backoff, HTTP range requests for resume, backup URLs, content-type validation, and DASH audio/video merging via ffmpeg. It is self-contained (imports only standard library, ffmpeg, and its own types) and could be extracted into a standalone library for any project needing robust media downloading.
Implements a cache factory that returns either a local in-memory cache or a Redis cache based on configuration, both adhering to an abstract cache interface. The pattern is clean, well-tested, and reusable in any Python application requiring a pluggable caching layer.
Provides an asynchronous file writer for CSV, JSON, and JSONL formats with thread-safe locking. It handles file opening, writing, and closing in an async context, making it a simple but reusable utility for any async Python project that needs to write structured data to disk.