Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
MechanicalSoup is a Python library for automating interaction with websites. It simulates a web browser by automatically storing and sending cookies, following redirects, and submitting forms. It does not execute JavaScript, making it a lightweight alternative to Selenium for sites that rely on server-side logic. The library is built on top of Requests for HTTP sessions and BeautifulSoup for HTML parsing and manipulation. The architecture provides two main classes: Browser (low-level, stateless) and StatefulBrowser (high-level, stateful). StatefulBrowser is the recommended interface, maintaining the current page, URL, and selected form as internal state. Users can open URLs, follow links, select forms, fill in fields, and submit. The Form class handles form manipulation including text inputs, checkboxes, radio buttons, select dropdowns, textareas, and file uploads. It also correctly implements HTML form submission semantics such as disabled controls, fieldset disabling, and multipart encoding. Key entry points include StatefulBrowser.open(), select_form(), submit_selected(), follow_link(), and download_link(). The library depends on requests, beautifulsoup4, and lxml. It is designed as a reusable library, not a standalone application, and is typically installed via pip and imported into user scripts. The codebase is well-tested with 100% coverage, has comprehensive documentation, and follows semantic versioning.