Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
This repository is a curated, categorized directory of over 10,000 Apify actors (web scraping and automation APIs), organized into thematic folders (e.g., AI, Jobs, Real Estate, Videos) with affiliate tracking links. It is primarily a static reference catalog intended for integration with the OpenClaw AI assistant, but also includes two Node.js scripts that fetch actor metadata from the Apify Store API and regenerate the markdown files. The repo is a hybrid: the scripts form a simple data pipeline (fetch, filter, generate), while the bulk of the content is a manually curated list. The code quality is basic—scripts are functional but tightly coupled to the Apify API and output format. The value lies in the aggregation and categorization of APIs, not in the software craftsmanship. The list is large but contains duplicates, placeholder entries, and potential staleness due to reliance on third-party actor availability.