This repository provides a Python library for converting web pages into text optimized for Large Language Model (LLM) input. It fetches HTML content using Playwright (headless browser) and processes it to extract meaningful text while preserving structural information like tables, links, and images in a text format. The core processing logic converts HTML tables to markdown, handles image and link preservation (or removal), and strips unnecessary tags (script, style) to reduce noise. The library is designed as a preprocessing step before feeding web content to LLMs for tasks such as data extraction or summarization. It includes three main modules: HTML fetching, HTML processing, and a combined convenience function.
This repository is the official CLI tool for Firecrawl, a web scraping, crawling, and structured data extraction platform. Built with Node.js and TypeScript, it provides a comprehensive command-line interface (CLI) using the Commander framework. In production, it allows users to scrape individual URLs, perform multi-page crawls, search the web (text, images, news), map site structures, interact with cloud browser sessions, create recurring content monitors, and use AI agents for complex extraction tasks. It also includes a research paper search feature covering ~43 million abstracts from PubMed, bioRxiv, medRxiv, and arXiv, as well as a developer index for searching GitHub issues, PRs, and READMEs. The CLI is designed for both direct human use and integration with AI coding agents via MCP (Model Context Protocol) and a skills system. Architecturally, it follows a command-per-file pattern in src/commands/, with shared utilities in src/utils/ for API client management, configuration/credential handling, output formatting, authentication (PKCE OAuth, API key, keyless free tier), and job polling. The core data flow is: user input -> command handler -> Firecrawl REST API (cloud or self-hosted) -> formatted output to stdout or file. External dependencies include the firecrawl SDK, commander, @inquirer/prompts, yaml, and zod-to-json-schema. The tool supports binary builds via Bun compile and is distributed via npm, Homebrew, and GitHub Releases with .deb/.rpm packages. It also includes interactive setup wizards, MCP server installation for coding agents, and a native skill installer. While primarily an application, the research paper search feature gives it a hybrid data-access character, but it does not contain data pipelines, notebooks, or analytical models. The codebase is well-tested (Vitest) and actively maintained, with CI/CD workflows for testing, npm publishing, and binary releases. However, the large command surface and agent-specific integrations introduce some maintenance complexity, and the reliance on the firecrawl SDK and external APIs means changes upstream could require updates.