Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
MechanicalSoup is a Python library that automates website interaction without executing JavaScript. It wraps requests.Session for HTTP transport and BeautifulSoup for HTML parsing, providing a mechanize-like API with two primary classes: Browser (stateless, low-level) and StatefulBrowser (maintains page, URL, and selected form state). The code base is compact (four core modules) and behavior is deliberately browser-faithful: disabled form controls (including those inside disabled fieldsets) are not submitted, the first option of a select without a selection is sent, enctype handling matches real browsers, and the Referer header is managed per HTML semantics. A notable security fix remediated CVE-2023-34457 by requiring explicit file-object opening instead of accepting path strings for multipart uploads. The test suite is extensive, runs at 100% coverage, and includes edge cases for encoding detection, duplicate submit buttons, form-associated external elements, and header overriding.