$ show firecrawl/cli
This repository is the official CLI tool for Firecrawl, a web scraping, crawling, and structured data extraction platform. Built with Node.js and TypeScript, it provides a comprehensive command-line interface (CLI) using the Commander framework. In production, it allows users to scrape individual URLs, perform multi-page crawls, search the web (text, images, news), map site structures, interact with cloud browser sessions, create recurring content monitors, and use AI agents for complex extraction tasks. It also includes a research paper search feature covering ~43 million abstracts from PubMed, bioRxiv, medRxiv, and arXiv, as well as a developer index for searching GitHub issues, PRs, and READMEs. The CLI is designed for both direct human use and integration with AI coding agents via MCP (Model Context Protocol) and a skills system. Architecturally, it follows a command-per-file pattern in src/commands/, with shared utilities in src/utils/ for API client management, configuration/credential handling, output formatting, authentication (PKCE OAuth, API key, keyless free tier), and job polling. The core data flow is: user input -> command handler -> Firecrawl REST API (cloud or self-hosted) -> formatted output to stdout or file. External dependencies include the firecrawl SDK, commander, @inquirer/prompts, yaml, and zod-to-json-schema. The tool supports binary builds via Bun compile and is distributed via npm, Homebrew, and GitHub Releases with .deb/.rpm packages. It also includes interactive setup wizards, MCP server installation for coding agents, and a native skill installer. While primarily an application, the research paper search feature gives it a hybrid data-access character, but it does not contain data pipelines, notebooks, or analytical models. The codebase is well-tested (Vitest) and actively maintained, with CI/CD workflows for testing, npm publishing, and binary releases. However, the large command surface and agent-specific integrations introduce some maintenance complexity, and the reliance on the firecrawl SDK and external APIs means changes upstream could require updates.
The LLM identified 3 files worth highlighting for reuse:
Provides a singleton Firecrawl SDK client with automatic keyless mode detection and helpers (keylessRequest, keylessGet). Encapsulates instantiation, reuse of credentials, and API URL configuration. Could be extracted and reused by any project consuming the Firecrawl API, with minimal adaptation needed (just import and call getClient).
Implements writeOutput for writing to stdout or file, and handleScrapeOutput for rendering scrape results in multiple formats (raw, JSON) based on user options. Decoupled from command logic, making it reusable across different CLI tools that need flexible output handling. Demonstrates clean separation of presentation from business logic.
Handles loading, validating, and persisting configuration (YAML) with support for default values, global config paths, and schema validation. Designed to be extensible for any CLI tool requiring persistent configuration. The modular validation and path resolution make it a strong candidate for extraction into a standalone configuration utility.