$ show D4Vinci/Scrapling
Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
The LLM identified 4 files worth highlighting for reuse:
Implements a ProxyRotator class with pluggable rotation strategies (cyclic by default, supports random, weighted, and custom callables). Thread-safe via a lock, supports per-request override, and integrates seamlessly with both HTTP and browser sessions. The design is isolated and could be extracted into a standalone proxy rotation utility for any scraping project.
Provides CheckpointManager for pause/resume functionality in crawls. Uses atomic file operations (write to temp file, then rename) to prevent corruption. Stores serialized scheduler state (pending requests, stats) and can be restored on restart. The pattern is reusable for any long-running batch process that needs fault tolerance.
ResponseFactory class that normalizes responses from different sources (curl_cffi HTTP responses, Playwright page responses) into a unified Response object. Handles encoding detection, charset extraction, and body conversion. This factory pattern is reusable for any project that needs to abstract over multiple HTTP client libraries.
StorageSystemMixin and SQLiteStorageSystem provide the backend for adaptive element tracking. Stores element properties (tag, attributes, text, position) keyed by domain and selector, enabling relocation after website changes. The design supports custom storage backends (e.g., Redis) and is isolated enough to be reused in other scraping frameworks.