Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
Parklife is a Ruby library and CLI tool that converts Rack-based web applications (like Rails or Sinatra) into static HTML sites suitable for deployment on static hosting platforms (Netlify, Vercel, GitHub Pages, S3, etc.). It works by crawling the application's routes starting from defined entry points, making HTTP requests to each endpoint via Rack::Test, and saving the responses as appropriate files in a build directory. The library intelligently handles different HTTP response types through a responder pattern (200=OK, 304=NotModified, 404=NotFound, etc.), supports automatic link discovery and crawling from HTML responses, and provides flexible configuration for output formatting, caching via ETags, and 404 handling. Key features include route crawling with depth control, intelligent file naming logic (HTML vs other content types), and multiple reporter formats (progress dots, detailed logs, or silent).
WaterCrawl is a full-stack web crawling and data extraction platform, built as a SaaS-style multi-tenant application. The backend is implemented in Django 5.1 with Django REST Framework, providing REST APIs for crawl requests (single/batch URL), web search (Google Custom Search integration), sitemap generation, and proxy server management. Asynchronous task execution is handled by Celery workers that invoke Scrapy spiders (SiteScrapper, GoogleCustomSearchScrapper, SitemapScrapper) and optionally use Playwright for JavaScript rendering. Real-time status updates are pushed to the frontend via Redis Pub/Sub and Server-Sent Events (SSE). The frontend is a React SPA (with TypeScript, Tailwind CSS, react-hook-form) that provides dashboard pages for configuring and monitoring crawl/search/sitemap jobs, managing teams and API keys, viewing usage statistics, and handling subscription billing through Stripe. Authentication supports JWT (simplejwt) and OAuth (Google, GitHub), with email verification and team invitation flows. The application stores extracted content (markdown, HTML, attachments) in MinIO or S3-compatible storage, and metadata in PostgreSQL.