Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
Puppeteer is a high-level Node.js library for browser automation, primarily targeting Chrome/Chromium and Firefox. It provides a rich API to control browsers programmatically via the Chrome DevTools Protocol (CDP) and WebDriver BiDi, supporting both headless and headed modes. The library is organized as a monorepo with multiple packages: `puppeteer` (full library with browser management), `puppeteer-core` (core automation API without browser download), `@puppeteer/browsers` (browser download and CLI), `@puppeteer/ng-schematics` (Angular integration), and a test server. The architecture features abstract API classes (Page, Browser, ElementHandle) with separate CDP and BiDi implementations, a custom event system, and extensive use of TypeScript generics and resource management patterns. Puppeteer is widely used for end-to-end testing, web scraping, and automation tasks, with a strong emphasis on cross-browser compatibility and modern web platform features.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
HeadlessX is a self-hosted web scraping and browser automation platform with anti-detection capabilities. It provides a Next.js 16 dashboard, an Express 5 API with Prisma/PostgreSQL, a BullMQ queue worker, and a published CLI. The core value proposition is anti-detection scraping via the headfox-js package (Camoufox-compatible Firefox bundles), fingerprint generation, CAPTCHA solving, and LLM-friendly markdown output. The system supports multiple operators: website scraping/crawling/mapping, Google AI Search, Tavily search, Exa search, and YouTube extraction. It includes sidecar services for HTML-to-Markdown (Go) and YouTube extraction (Python FastAPI). The architecture is a monorepo with Nx orchestration, Docker Compose deployment, and an MCP endpoint for AI assistant integration. External dependencies include Redis, Postgres, Playwright, MaxMind GeoIP, and external APIs (Tavily, Exa, Google). The codebase is well-structured with clear separation of concerns, modern TypeScript, and reusable patterns in both frontend and backend.
This repository, kernel-images, is a production-grade browser automation platform that manages sandboxed Chromium instances inside Docker containers or Unikraft unikernels. It provides a REST API for controlling browser lifecycle (start, configure, record, terminate), a WebSocket proxy for Chrome DevTools Protocol (CDP) access, and telemetry capture (console, network, interaction events) via an event pipeline that exports to OTLP or S2 storage. The system supports headful and headless modes, with a Vue.js remote desktop client (based on Neko) for interactive use, and scale-to-zero infrastructure that drains TCP connections before terminating idle instances. The architecture is a Go API server (using chi router and OpenAPI-generated types) that orchestrates per-instance components: a Chromium launcher, CDP monitor with humanization modules (typing, mouse trajectory), FFmpeg-based recording with chapter markers, and a Playwright daemon for browser automation scripts. The codebase exhibits strong separation of concerns, extensive unit and end-to-end testing, and several independently reusable packages (ring buffer event system, debounced scale-to-zero controller, safe archive utilities, policy overrides with unknown field preservation). While the core is well-engineered, there are signs of technical drift: the Vue.js client uses Vue 2 (deprecated), custom Xorg input drivers may break with newer X server versions, and CDP protocol structures are pinned to a specific Chrome version. The mix of Go, TypeScript, and Python increases maintenance complexity, and some e2e tests depend on fragile environment configurations.