$ show saifyxpro/HeadlessX
HeadlessX is a self-hosted web scraping and browser automation platform with anti-detection capabilities. It provides a Next.js 16 dashboard, an Express 5 API with Prisma/PostgreSQL, a BullMQ queue worker, and a published CLI. The core value proposition is anti-detection scraping via the headfox-js package (Camoufox-compatible Firefox bundles), fingerprint generation, CAPTCHA solving, and LLM-friendly markdown output. The system supports multiple operators: website scraping/crawling/mapping, Google AI Search, Tavily search, Exa search, and YouTube extraction. It includes sidecar services for HTML-to-Markdown (Go) and YouTube extraction (Python FastAPI). The architecture is a monorepo with Nx orchestration, Docker Compose deployment, and an MCP endpoint for AI assistant integration. External dependencies include Redis, Postgres, Playwright, MaxMind GeoIP, and external APIs (Tavily, Exa, Google). The codebase is well-structured with clear separation of concerns, modern TypeScript, and reusable patterns in both frontend and backend.
The LLM identified 2 files worth highlighting for reuse:
Central utility for configuring Playwright launch options from a fingerprint, proxy, GeoIP, locale, fonts, WebGL, addons, and environment variables. This is the core anti-detection logic that can be extracted and reused in any Playwright-based scraping project to avoid bot detection. The function is self-contained, parameterized, and demonstrates deep knowledge of browser fingerprinting and stealth techniques.
Registration pattern for MCP (Model Context Protocol) tools that exposes scraping, search, and job management capabilities to AI assistants. It defines Zod schemas, tool handlers, and error/success response formatting. This pattern is reusable across any Node.js project that needs to expose arbitrary tools to LLMs via the MCP standard, decoupled from the specific business logic in the tool implementations.