Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
This repository provides a stealth browser automation MCP (Model Context Protocol) server that offers undetectable control over real browser instances (Chrome, Chromium, Edge) via the Chrome DevTools Protocol (CDP) and the `nodriver` library. It exposes 97 tools across 11 sections including navigation, DOM interaction, element extraction (styles, structure, events, animations), network interception and modification (including dynamic Python-powered hooks), CDP command execution, cookie management, tab control, and proxy forwarding with authentication. The architecture is a FastMCP-based server with singleton managers for browser instances, network interception, element cloning, and process cleanup. It supports stdio and HTTP transports, Docker deployment, and optional bearer auth. The system is designed to bypass anti-bot detection (Cloudflare, CreepJS, login walls) and is used for scraping, testing, and AI agent automation. While the codebase is well-structured and the functionality is comprehensive, there is some code duplication across multiple element cloner implementations and technical drift risk due to tight coupling with CDP APIs and the `py2js` transpiler dependency.
Botright is a Python library built on Playwright that provides advanced browser automation with stealth features to evade bot detection. It includes proxy management, fingerprint spoofing using real browser profiles (chrome-fingerprints), human-like mouse movements, and CAPTCHA solving for hCaptcha, reCAPTCHA, and Geetest (though Geetest support is currently disabled). The library wraps Playwright's API with a custom layer that intercepts browser events, modifies user agent, platform, and other fingerprinting vectors, and injects anti-detection scripts. It supports both headless and headed modes, proxy rotation, and response caching. The codebase is well-structured but has commented-out modules indicating incomplete transitions (e.g., Geetest).