$ show unclecode/crawl4ai
Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
The LLM identified 4 files worth highlighting for reuse:
Provides a process-wide egress proxy that intercepts all outbound HTTP connections from the library's own clients (httpx, aiohttp). Uses a single rule (`not ip.is_global`) to block internal network requests, preventing SSRF attacks. The `resolve_and_pin` method resolves hostnames to IPs and pins them to prevent DNS rebinding. This component is self-contained and can be extracted into any Python application needing to enforce outbound network restrictions.
Implements `MemoryAdaptiveDispatcher` and `SemaphoreDispatcher` for concurrent URL crawling with resource-aware rate limiting. The memory-adaptive variant monitors system memory and adjusts concurrency dynamically, while supporting streaming of results via `arun_many(stream=True)`. This pattern is reusable for any async batch processing task that needs to respect system constraints and provide real-time output.
Defines the abstract `DeepCrawlStrategy` base class with a `ContextVar`-based cancellation mechanism, state export/resume (`export_state`, `resume_state`), and pluggable `FilterChain` and `URLScorer`. Concrete strategies (BFS, DFS, Best-First) extend this base. The pattern is reusable for any recursive crawling or graph traversal that requires cancellation, state persistence, and scoring.
Compiles C4A-Script (a domain-specific language for browser automation) into JavaScript using a Lark grammar. Employs a `Result` pattern (no exceptions) with error code mapping and supports LLM-based script generation from natural language descriptions. This component is self-contained and can be extracted to build DSL compilers for other automation tasks.