Firecrawl is a production-grade web scraping, crawling, and structured data extraction platform designed to feed AI agents and LLMs with clean, markdown-formatted web data. The system exposes a multi-version REST API (v0, v1, v2) with endpoints for scraping single URLs, crawling entire sites, mapping site structures, searching the web, extracting structured JSON, monitoring page changes, and running autonomous AI agents. It supports multiple output formats including markdown, HTML, screenshots, PDF parsing, product/menu extraction, and branding detection. The architecture is a microservices-style monolith with a Node.js/Express API server at its core, backed by Redis for job queues and caching, PostgreSQL for persistent state, ClickHouse for analytics, Bigtable for job access logs, and GCS for blob storage. Background workers handle crawl orchestration, PDF processing, webhook delivery, and billing via RabbitMQ and FoundationDB queues. The platform integrates with external services for threat protection (Google Web Risk, Zscaler), billing (Autumn), search (SearXNG, DuckDuckGo, X Search), and LLM-based extraction (OpenAI, Anthropic). Nine official SDKs (Python, JavaScript/TypeScript, Go, Java, .NET, Ruby, PHP, Rust, Elixir) wrap the API with typed models, error handling, and async support, making it accessible across ecosystems. The codebase is extensive and well-tested, with over 2,000 files spanning controllers, scrapers, transformers, services, and utilities, but shows signs of technical drift through deprecated v0/v1 endpoints, multiple parallel implementations of similar logic (e.g., extract pipelines), and configuration sprawl across environment variables.

This repository is the official CLI tool for Firecrawl, a web scraping, crawling, and structured data extraction platform. Built with Node.js and TypeScript, it provides a comprehensive command-line interface (CLI) using the Commander framework. In production, it allows users to scrape individual URLs, perform multi-page crawls, search the web (text, images, news), map site structures, interact with cloud browser sessions, create recurring content monitors, and use AI agents for complex extraction tasks. It also includes a research paper search feature covering ~43 million abstracts from PubMed, bioRxiv, medRxiv, and arXiv, as well as a developer index for searching GitHub issues, PRs, and READMEs. The CLI is designed for both direct human use and integration with AI coding agents via MCP (Model Context Protocol) and a skills system. Architecturally, it follows a command-per-file pattern in src/commands/, with shared utilities in src/utils/ for API client management, configuration/credential handling, output formatting, authentication (PKCE OAuth, API key, keyless free tier), and job polling. The core data flow is: user input -> command handler -> Firecrawl REST API (cloud or self-hosted) -> formatted output to stdout or file. External dependencies include the firecrawl SDK, commander, @inquirer/prompts, yaml, and zod-to-json-schema. The tool supports binary builds via Bun compile and is distributed via npm, Homebrew, and GitHub Releases with .deb/.rpm packages. It also includes interactive setup wizards, MCP server installation for coding agents, and a native skill installer. While primarily an application, the research paper search feature gives it a hybrid data-access character, but it does not contain data pipelines, notebooks, or analytical models. The codebase is well-tested (Vitest) and actively maintained, with CI/CD workflows for testing, npm publishing, and binary releases. However, the large command surface and agent-specific integrations introduce some maintenance complexity, and the reliance on the firecrawl SDK and external APIs means changes upstream could require updates.