Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
Agent Reach is a Python CLI tool that serves as a capability layer for AI agents, giving them one-click access to ~15 internet platforms (Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, LinkedIn, etc.) without requiring manual setup. It does not wrap these platforms itself; instead, it selects, installs, health-checks, and routes to the most reliable upstream CLI tools (e.g., twitter-cli, yt-dlp, gh CLI, OpenCLI, mcporter) for each platform. The architecture is built around a multi-backend channel pattern: each platform has an ordered list of backends (primary + fallbacks), and the `doctor` command probes each candidate in order to determine the active backend. This design handles upstream tool instability gracefully by allowing fallback routing without code changes. The codebase demonstrates strong engineering practices: a clean Channel ABC with `check()` and `can_handle()` contracts, a robust probe system that distinguishes missing/broken/timeout/error by actual execution, security-hardened file operations (atomic private writes, symlink rejection, bounded reads), and SSRF protection in URL normalization. The CLI provides commands for installation (safe-by-default, read-only mode), diagnostics, configuration, and skill registration for agent environments (Claude Code, OpenClaw, Cursor, OpenCode). An optional MCP server exposes read-only status. Agent Reach is purely an application — there are no notebooks, data pipelines, or ML components. Its value lies in reducing the friction of giving agents internet access, with a focus on zero-config defaults for public platforms and cookie-based auth for login-required ones. The project is actively maintained, with documented responses to upstream API changes (e.g., Bilibili blocking yt-dlp, Twitter API pay-per-use).
Firecrawl is a production-grade web scraping, crawling, and structured data extraction platform designed to feed AI agents and LLMs with clean, markdown-formatted web data. The system exposes a multi-version REST API (v0, v1, v2) with endpoints for scraping single URLs, crawling entire sites, mapping site structures, searching the web, extracting structured JSON, monitoring page changes, and running autonomous AI agents. It supports multiple output formats including markdown, HTML, screenshots, PDF parsing, product/menu extraction, and branding detection. The architecture is a microservices-style monolith with a Node.js/Express API server at its core, backed by Redis for job queues and caching, PostgreSQL for persistent state, ClickHouse for analytics, Bigtable for job access logs, and GCS for blob storage. Background workers handle crawl orchestration, PDF processing, webhook delivery, and billing via RabbitMQ and FoundationDB queues. The platform integrates with external services for threat protection (Google Web Risk, Zscaler), billing (Autumn), search (SearXNG, DuckDuckGo, X Search), and LLM-based extraction (OpenAI, Anthropic). Nine official SDKs (Python, JavaScript/TypeScript, Go, Java, .NET, Ruby, PHP, Rust, Elixir) wrap the API with typed models, error handling, and async support, making it accessible across ecosystems. The codebase is extensive and well-tested, with over 2,000 files spanning controllers, scrapers, transformers, services, and utilities, but shows signs of technical drift through deprecated v0/v1 endpoints, multiple parallel implementations of similar logic (e.g., extract pipelines), and configuration sprawl across environment variables.
ScrapeGraphAI is a Python library that extracts structured data from websites and local documents using LLM-driven graph pipelines. Instead of a traditional web scraper with hardcoded selectors, users define extraction through natural-language prompts and optional Pydantic schemas; the library then executes a directed graph of nodes such as fetching, parsing, chunking, and answer generation. The public API is a set of graph classes (SmartScraperGraph, SearchGraph, JSONScraperGraph, CSVScraperGraph, SpeechGraph, ScriptCreatorGraph, etc.) in scrapegraphai/graphs, each composing reusable nodes from scrapegraphai/nodes and running them on a BaseGraph execution engine. It is consumed programmatically in Python and supports many LLM providers (OpenAI, Ollama, Groq, Mistral, Anthropic, DeepSeek, xAI, NVIDIA, MiniMax) and multiple fetching backends (Playwright, Selenium, Chromium, BrowserBase, Scrape.do). The architecture is a lightweight DAG framework: nodes have typed inputs/outputs, can branch conditionally, iterate, merge results, and accumulate cost/token metadata; BaseGraph walks the graph, handles state, and collects execution information. The repository also contains document loaders, model token/cost registries, prompt templates, telemetry, proxy rotation, screenshot-scraping utilities, a GraphBuilder that constructs graphs from natural language, and integrations such as Burr and Indexify. While the README aggressively promotes a commercial hosted API at scrapegraphai.com, the repository itself is a self-contained open-source library with examples and tests. The codebase is generally well structured and modular, with clear separation between graphs, nodes, prompts, helpers, utils, and models. It shows signs of rapid growth: some typos, duplicated test fixtures, hardcoded model pricing tables, a Python version mismatch between pyproject and Dockerfile, and dependency fragility around LangChain and optional browser/OCR packages. These issues are manageable but give the codebase a medium technical drift. Overall it is an active, useful library for LLM-based extraction, and several utilities are good candidates for reuse outside the project.
feedgrab is a universal content aggregation tool that fetches, normalizes, and saves content from 18+ platforms (Twitter/X, Xiaohongshu, WeChat, YouTube, Bilibili, GitHub, HackerNews, Medium, Reddit, Weibo, Douyin, Telegram, RSS, Feishu, Kdocs, Zsxq, FlowUs, and more) into Obsidian-compatible Markdown files with YAML frontmatter. It operates as a CLI tool, a Python library, and an MCP server, featuring a modular architecture with per-platform fetchers that implement a multi-tier fallback strategy (direct API → CDP browser reuse → Playwright launch → Jina Reader) to maximize resilience against anti-scraping measures. The system includes robust utilities for deduplication, media download, Groq Whisper transcription, paywall bypass, and cookie rotation for authenticated platforms. Originally developed as a 'vibe coding' personal project, it has evolved into a well-structured application with a service layer, typed configuration, and comprehensive CLI commands (login, doctor, batch search, setup wizard). It is actively maintained, with frequent commits addressing platform API changes and adding new features.
The ultimate-web-scraper repository is a comprehensive PHP toolkit for web scraping and related web interaction tasks. It provides a robust HTTP client (WebBrowser) that handles cookies, redirects, SSL/TLS, and proxy support, enabling reliable web requests. For HTML processing, it includes the TagFilter library, which offers high-performance parsing with CSS3 selector support, streaming capabilities for large files, and built-in XSS prevention. The toolkit also features WebSocket client and server implementations, a simple HTTP server, and utilities for asynchronous operations, DNS over HTTPS, and data streaming. Designed as a modular library, it allows developers to extract and reuse components like the HTTP client, HTML parser, or async helper in other projects with minimal dependencies.
This repository implements a production-grade Google Maps business data extraction platform with multiple execution modes: CLI, web UI, REST API, AWS Lambda, and a self-hosted SaaS edition. The core scraping engine uses Playwright for browser automation and also supports a stealth HTTP fast mode. It extracts 36+ structured fields per business (name, address, phone, website, reviews, ratings, emails, coordinates, hours, etc.) and provides deduplication, grid-based area coverage, email extraction via website visits, and review parsing. The architecture is modular with packages for scraping jobs, runners (file, database, web, Lambda, SaaS worker), output writers (CSV, JSON, PostgreSQL, S3, custom plugins), job queuing via River, and a comprehensive admin dashboard with cloud provisioning on DigitalOcean and Hetzner. The system is designed for both single-user CLI use and multi-tenant SaaS deployments with authentication, 2FA, API key management, and scalable worker pools. The codebase demonstrates strong Go engineering practices with well-defined interfaces, dependency injection, and test coverage. Notable reusable components include cryptography utilities, HTTP server middleware, rate limiting, and a CLI prompt library. However, the repository has some maintenance concerns: two separate database migration sets for legacy and SaaS modes, in-memory deduplication that doesn't persist across restarts, committed test SSH keys in the `local/` directory, and a large dependency surface that could introduce version conflicts. The scraper logic is inherently brittle due to reliance on Google Maps internal data structures, which are subject to change without notice. For AI agent consumption: This is a powerful off-the-shelf scraping tool that can be deployed as a Docker container, AWS Lambda, or managed SaaS. It provides a complete API and web interface for managing scraping jobs, monitoring progress, and exporting data. The SaaS edition includes worker auto-provisioning, making it suitable for large-scale lead generation operations. The code is actively maintained with recent commits addressing performance and feature enhancements.
This repository implements a web scraper for the EastMoney stock forum (guba.eastmoney.com) using Selenium and MongoDB. It provides two main crawlers: PostCrawler scrapes post listings (title, views, comment count, URL, date, author) from paginated forum pages, and CommentCrawler scrapes individual post pages for comments (content, likes, date, sub-comments). The scraper uses a stealth.js script to evade bot detection, handles errors by restarting the WebDriver, and supports multi-threaded crawling for multiple stock symbols. Data is stored in MongoDB collections named post_<symbol> and comment_<symbol>. The code is a monolithic script with classes for crawling, parsing, and database operations. It is a functional but unremarkable scraper suitable for small-scale data collection, with basic error handling and no testing or dependency management.
This is a Go CLI tool that scrapes certification exam questions and answers from the examtopics.com website, which is otherwise paywalled or requires manual browsing. The tool accepts a provider (e.g., google, cisco) and a search string to filter exams, then fetches all discussion pages, extracts question data (text, choices, community-voted answer, timestamp, comments), and outputs the results in Markdown, JSON, HTML, PDF, or plain text. It includes optional caching via a GitHub repository of pre-scraped data, rate limiting, retry logic, and concurrent request concurrency. The architecture is a single-binary monolith with internal packages for fetching, scraping, and file output. The code is functional but unremarkable—standard Go patterns with some concurrency, though the global counter and finalizer usage are minor anti-patterns.
This repository is an automated IPTV channel source scraper and validator, primarily targeting Chinese hotel and groupcast (组播) streams. The system uses Selenium with headless Chrome to scrape search engines like FOFA and ZoomEye for IPTV-related URLs, then brute-force scans IP ranges to discover accessible streaming servers. It downloads sample TS segments from discovered m3u8 streams to measure download speed and optionally validate resolution using OpenCV. The output is a curated playlist in both CSV and M3U formats (itvlist.txt, itvlist.m3u) organized into categories like 央视频道, 卫视频道, and 其他频道. The project includes multiple execution scripts (itv.py, new.py, itv_all.py, ZHGXTV.py) and a GitHub Actions workflow for daily automated runs. While functional, the codebase suffers from hardcoded IP lists, fragile error handling, tight coupling, and reliance on external services that may change. It is a pragmatic tool rather than a polished application.