$ show iBigQiang/feedgrab
feedgrab is a universal content aggregation tool that fetches, normalizes, and saves content from 18+ platforms (Twitter/X, Xiaohongshu, WeChat, YouTube, Bilibili, GitHub, HackerNews, Medium, Reddit, Weibo, Douyin, Telegram, RSS, Feishu, Kdocs, Zsxq, FlowUs, and more) into Obsidian-compatible Markdown files with YAML frontmatter. It operates as a CLI tool, a Python library, and an MCP server, featuring a modular architecture with per-platform fetchers that implement a multi-tier fallback strategy (direct API → CDP browser reuse → Playwright launch → Jina Reader) to maximize resilience against anti-scraping measures. The system includes robust utilities for deduplication, media download, Groq Whisper transcription, paywall bypass, and cookie rotation for authenticated platforms. Originally developed as a 'vibe coding' personal project, it has evolved into a well-structured application with a service layer, typed configuration, and comprehensive CLI commands (login, doctor, batch search, setup wizard). It is actively maintained, with frequent commits addressing platform API changes and adding new features.
The LLM identified 6 files worth highlighting for reuse:
Unified HTTP client that wraps curl_cffi with Chrome TLS impersonation, automatic fallback to requests, single session management, proxy support, and configurable timeouts. Can be extracted as a standalone library for any project needing resilient HTTP requests with anti-detection capabilities.
File-based global dedup index with load_index, has_item, add_item, and save_index functions. Supports old-format migration and platform-specific item_id generation. A reusable pattern for any batch processing pipeline that needs to avoid reprocessing items across runs.
Downloads media files (images, videos) to local attachments directory, replaces original URLs with relative Markdown paths, and optimizes image URLs. Can be extracted as a standalone media downloader with parallel downloads and path resolution.
Extracts Schema.org JSON-LD article bodies from HTML pages, supporting multiple JSON-LD structures and text concatenation. A reusable utility for any project that needs to parse structured data from web pages for clean content extraction.
Reusable stealth browser engine providing stealth_launch(), get_stealth_context_options(), setup_resource_blocking(), and generate_referer(). Configures Chromium with anti-detection flags, resource blocking, and persistent contexts. Can be extracted as a standalone Playwright stealth bootstrapper for any scraping project.
Manages Twitter/X cookie loading from multiple sources (env vars, JSON files, Chrome CDP), implements account rotation with rate-limit cooldown tracking, and provides fetch_with_cookie_rotation helper. A reusable pattern for any platform requiring multi-account authentication and graceful fallback under rate limits.