Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
WaterCrawl is a full-stack web crawling and data extraction platform, built as a SaaS-style multi-tenant application. The backend is implemented in Django 5.1 with Django REST Framework, providing REST APIs for crawl requests (single/batch URL), web search (Google Custom Search integration), sitemap generation, and proxy server management. Asynchronous task execution is handled by Celery workers that invoke Scrapy spiders (SiteScrapper, GoogleCustomSearchScrapper, SitemapScrapper) and optionally use Playwright for JavaScript rendering. Real-time status updates are pushed to the frontend via Redis Pub/Sub and Server-Sent Events (SSE). The frontend is a React SPA (with TypeScript, Tailwind CSS, react-hook-form) that provides dashboard pages for configuring and monitoring crawl/search/sitemap jobs, managing teams and API keys, viewing usage statistics, and handling subscription billing through Stripe. Authentication supports JWT (simplejwt) and OAuth (Google, GitHub), with email verification and team invitation flows. The application stores extracted content (markdown, HTML, attachments) in MinIO or S3-compatible storage, and metadata in PostgreSQL.