Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
This is a full-stack web application for performing open-source intelligence (OSINT) analysis on any website. The system runs 40+ checks encompassing DNS, SSL/TLS, HTTP security headers, mail configuration (SPF/DKIM/DMARC), social media presence, trackers, blocklists, threat feeds, carbon footprint, tech stack, and more. The frontend is built with Astro and React, featuring a reactive results view with a masonry grid, advisory panel, and progress bar. The backend is a Node.js Express server (also serverless-friendly) with individual API handlers per check, wrapped in a common middleware layer for error handling, rate limiting, and skip logic. The application is designed for self-hosting and supports deployment to Vercel, Netlify, Docker, Render, and Fly.io. It relies on many external sources (DNS, WHOIS, SSL Labs, HIBP, Google Safe Browsing, Shodan, etc.) and uses Puppeteer/Chromium for headless browser tasks. The codebase demonstrates a clean modular architecture, reusable UI components, and a novel analysis rule system that transforms raw data into structured findings with severity levels.
Asciidoctor Web PDF is a CLI tool that converts AsciiDoc documents to PDF using web technologies. It employs a multi-stage pipeline: Asciidoctor converts .adoc to HTML, Puppeteer renders the HTML via the Vivliostyle Viewer (CSS paged media), and pdf-lib adds PDF outline/bookmarks and metadata. The architecture features clean separation of concerns with dedicated modules for browser management, MathJax processing, syntax highlighting, outline generation, and metadata injection. Notable for its sophisticated handling of Node.js SEA binary execution paths and server-side MathJax rendering with proper async font loading.
Memlab is a comprehensive JavaScript memory leak detection and heap analysis framework developed by Meta. It provides a CLI, a programmatic API, and an MCP (Model Context Protocol) server for AI-assisted debugging. The system captures heap snapshots from Chromium (via Puppeteer), Node.js, Electron.js, and Hermes, then parses and analyzes them to identify memory leaks, optimization opportunities, and retention patterns. It includes specialized detectors for detached DOM elements, React component leaks, event listener accumulation, duplicate strings, and global variable bloat. The architecture is a monorepo with packages for core parsing and algorithms, E2E browser automation, heap analysis plugins, a public API, a CLI, an in-browser visualizer (MemLens), and an MCP server that exposes over 100 analysis tools. Key data flows involve warmup, browser interaction, snapshot capture, parsing, leak filtering, trace clustering (heuristic and ML-based), and report generation. External dependencies include Puppeteer, Chromium, and the MCP SDK.
Shower is an HTML presentation engine built as a monorepo containing a core library, two CSS-only themes (Ribbon and Material), a CLI tool, and a starter template. The core engine, written in vanilla JavaScript, handles slide management, keyboard and touch navigation, inner slide steps, progress bar, timer, full-screen support, and accessibility. The CLI provides commands for creating, serving, bundling, archiving, and publishing presentations, as well as PDF export via a self-downloading Chrome engine. Themes use advanced CSS features like custom properties and the tan(atan2()) scaling trick, and are entirely decoupled from the engine. The repository is well-tested with Playwright and GitHub Actions CI, uses modern Node.js (ESM, workspaces), and demonstrates clean architecture with lazy-loaded CLI commands and a plugin-based core.
This repository implements an MCP (Model Context Protocol) server that exposes Chrome DevTools capabilities as tools for AI coding assistants. In production, it launches a headless or headed Chrome browser via Puppeteer, connects to the DevTools frontend SDK, and provides over 50 tools covering navigation, input automation, network inspection, performance tracing, heap snapshot analysis, CSS inspection, accessibility auditing (via Lighthouse), PWA management, and extension debugging. The architecture is a Node.js server that runs either as a long-lived daemon (Unix socket IPC) or as a CLI tool, with a configuration pipeline supporting CLI flags, config files, and environment variables. Key entry points are `src/bin/chrome-devtools-mcp.ts` (MCP server) and `src/bin/chrome-devtools.ts` (standalone CLI). Data flows involve collecting console messages, network requests, and DevTools issues per page via `PageCollector`, processing performance traces via `ChunkedTraceParser` and `PerformanceTrace`, and formatting results through specialized formatters (`ConsoleFormatter`, `CssFormatter`, `HeapSnapshotFormatter`, etc.) that support both text and structured JSON output. External dependencies include Puppeteer for browser control, the DevTools frontend SDK for advanced features (heap snapshots, stack traces), Lighthouse for audits, and a telemetry system (Clearcut) for usage statistics.
mermaid-cli is a command-line interface for generating diagrams from Mermaid syntax. It processes input files (.mmd, .md, .markdown) containing Mermaid diagram definitions and outputs SVG, PNG, or PDF files. The architecture centers around a Puppeteer-driven rendering pipeline that launches a headless browser to render Mermaid diagrams via the Mermaid library. Key features include Markdown diagram extraction, configurable theming, CSS injection, icon pack support, and programmable Node.js API. The CLI entry point is src/cli.js, which delegates to the run() and renderMermaid() functions in src/index.js for core functionality. External dependencies include Puppeteer, Mermaid, and various layout engines (ELK, tidy-tree).
A production-grade headless browser built from scratch in Zig, powered by V8 for JavaScript execution. It implements a vast subset of Web Platform APIs (DOM, CSSOM, HTML/SVG elements, canvas, WebGL, IndexedDB, workers, fetch, WebSocket, etc.) and exposes them through Chrome DevTools Protocol (CDP), WebDriver BiDi, and MCP (Model Context Protocol) servers. Includes a built-in LLM agent mode for natural-language-driven automation, a scripting system (PandaScript) for deterministic replay, and a full network stack with ad blocking, rate limiting, caching, and TLS. Designed for AI agents and automation workloads, it claims significant performance advantages over Chromium in startup time and resource usage. Architecturally, the browser is a monolithic server binary with distinct internal modules: a browser core managing pages, frames, and V8 isolates; a network layer using libcurl with connection pooling and request deduplication; protocol servers (CDP, BiDi, MCP) communicating over WebSocket or HTTP; and an agent subsystem that wraps LLM APIs for task planning and execution. The codebase is heavily tested with hundreds of hand-written HTML conformance tests and runs Web Platform Tests via CI. The engineering demonstrates high craftsmanship: custom arena allocators with V8 memory accounting, compile-time prototype hierarchy reflection for Web IDL bindings, a GCRA rate limiter, a singleflight pattern for request deduplication, and a generic live-collection iterator. While some APIs are stubbed or partially implemented (e.g., WebGL always returns null, video dimensions are hardcoded), the core functionality is extensive and spec-compliant where implemented. The project is actively maintained with nightly orderfile optimizations and regular CI.
Unlighthouse is a CLI tool and web dashboard that scans entire websites using Google Lighthouse, aggregating results across all pages and presenting them in a real-time, filterable dashboard. The architecture is a monorepo with separate packages for core scanning logic, CLI, CI integration, a Vue 3 client, and a small h3 server for serving reports and WebSocket updates. Key entry points include the `unlighthouse` CLI (via `cac`), the `unlighthouse-ci` CI command, and the `createUnlighthouse` function from `@unlighthouse/core`. Primary data flows: URL discovery (sitemap, robots.txt, crawling), parallel Lighthouse runs via `puppeteer-cluster`, streaming results to the client, and exporting reports (HTML, JSON, CSV, LHCI). External dependencies include Puppeteer (Chrome), Lighthouse, and optionally the CrUX API for historical field data. The codebase is well-structured, uses ESM, and has a modern build pipeline with Vite and Vitest. Some integrations (Nuxt, Vite, Webpack) are deprecated, but the core functionality remains actively maintained. The client UI is feature-rich, with dark mode, search, pagination, and detailed audit result views.
Pa11y is a mature, widely-used automated accessibility testing tool that runs via command-line interface or Node.js API. It drives a headless Chrome browser (via Puppeteer) to load web pages, execute optional user interactions (actions), and then run one or more accessibility test runners (HTML_CodeSniffer and axe-core) against the page. Results are reported in multiple formats (CLI, CSV, JSON, HTML, TSV) and can be filtered by severity, ignored rules, or threshold. The architecture is a monolith library with a thin CLI wrapper, but the runner and reporter systems are modular and extensible via npm packages or local files. Key entry points are `bin/pa11y.js` for CLI and `lib/pa11y.js` for programmatic use. The data flow: launch Chrome, navigate to URL, run actions, inject runner scripts into the page, evaluate accessibility, collect and process issues, output via selected reporter. External dependencies include Puppeteer, axe-core, @pa11y/html_codesniffer, commander, and several utility libraries. The codebase is well-structured, thoroughly tested (unit and integration), and follows modern Node.js practices with async/await and ES6+ features.
HeadlessX is a self-hosted web scraping and browser automation platform with anti-detection capabilities. It provides a Next.js 16 dashboard, an Express 5 API with Prisma/PostgreSQL, a BullMQ queue worker, and a published CLI. The core value proposition is anti-detection scraping via the headfox-js package (Camoufox-compatible Firefox bundles), fingerprint generation, CAPTCHA solving, and LLM-friendly markdown output. The system supports multiple operators: website scraping/crawling/mapping, Google AI Search, Tavily search, Exa search, and YouTube extraction. It includes sidecar services for HTML-to-Markdown (Go) and YouTube extraction (Python FastAPI). The architecture is a monorepo with Nx orchestration, Docker Compose deployment, and an MCP endpoint for AI assistant integration. External dependencies include Redis, Postgres, Playwright, MaxMind GeoIP, and external APIs (Tavily, Exa, Google). The codebase is well-structured with clear separation of concerns, modern TypeScript, and reusable patterns in both frontend and backend.
CloakBrowser is a commercial stealth browser automation library that wraps Playwright (Python, JavaScript, .NET) with a custom patched Chromium binary. Its core purpose is to evade bot detection by spoofing browser fingerprints, providing human-like interaction (Bezier mouse moves, realistic typing with typos, smooth scrolling), and managing licensed binary downloads with cryptographic verification. The architecture is a multi-language SDK sharing identical logic across Python, TypeScript, and C#, with a common human interaction engine, proxy/GeoIP integration, and license enforcement via denial files and server validation. Key entry points are the `launch()` and `humanize()` APIs, CLI diagnostics (`info`, `login`, `install`), and the `cloakserve` CDP multiplexer. External dependencies include Playwright/Puppeteer, MaxMind GeoLite2 database, Ed25519 signature verification (BouncyCastle), and a license validation server. The codebase demonstrates high engineering discipline with extensive cross-language parity tests and security-conscious binary distribution, but carries drift risks from triple-language duplication and reliance on Playwright internal APIs.
This is a Node.js CLI application (Docker containerized) that automatically logs into the Epic Games Store using device code authentication (OAuth2 device grant), discovers currently free games (weekly promotions and general store discounts), validates ownership via Epic's GraphQL APIs, and sends actionable checkout links to the user through a variety of notification channels (email, Discord, Telegram, Pushover, Gotify, Slack, Home Assistant, Bark, Ntfy, Apprise, webhook, and local browser open). The application uses Puppeteer with a stealth plugin to simulate a browser session for cookie management and authenticated API calls. It runs on a configurable cron schedule, with optional run-on-startup and one-shot modes. It includes a lightweight Express web server to handle device code login redirects and optionally exposes itself via localtunnel for remote accessibility. Configuration is JSON/JSON5 with full class-validator based schema validation and environment variable fallbacks.
This repository is a Heroku buildpack designed to enable Puppeteer (a Node.js library for controlling headless Chrome/Chromium) to run on Heroku dynos. It installs the necessary system-level dependencies (shared libraries, fonts) required for Chromium to function in Heroku's Linux container environment. The buildpack consists of two scripts: bin/detect, which simply returns 'Puppeteer' to signal applicability, and bin/compile, which handles the core logic of detecting the Heroku stack, installing appropriate APT packages via apt-get with caching, setting up environment variables (LD_LIBRARY_PATH, etc.), and rewriting pkg-config files to ensure the installed libraries are discoverable by the application. The buildpack demonstrates solid engineering practices for Heroku buildpacks, including stack-aware dependency caching to speed up subsequent builds, proper environment variable propagation for downstream buildpacks, and clean separation of concerns. While the functionality is niche and the code is not exceptionally innovative, it is well-structured, maintainable, and solves a specific problem effectively. The repository has been updated to support recent Heroku stacks (heroku-22, heroku-24, heroku-26), indicating ongoing maintenance.
This repository is a TypeScript monorepo for an AI-powered autonomous web browsing agent called 'open-browser'. It consists of three packages: core library (open-browser), CLI (@open-browser/cli), and sandbox (@open-browser/sandbox). The core library orchestrates an agent loop that uses Playwright to control a Chromium browser, extracts DOM state via CDP, sends it to an LLM (OpenAI, Anthropic, Google) which decides actions in structured JSON, and executes those actions via a CommandExecutor. The system supports vision screenshots, extended thinking, compact mode, planning, evaluation, replay recording, and MCP server integration. The CLI provides commands for running agents, interactive sessions, and direct browser manipulation. The sandbox provides resource-limited execution. Key entry points include the Agent class, Viewport browser session, CommandExecutor, and BridgeServer. External dependencies include Playwright, Vercel AI SDK, Zod, Turndown, and various LLM provider SDKs. The architecture follows a modular monorepo with clear separation of concerns, event-driven communication via EventHub, and a guard/watchdog pattern for browser lifecycle management. The codebase demonstrates strong engineering practices with extensive use of Zod schemas for structured outputs, a command catalog pattern, and comprehensive telemetry and cost tracking.
This repository provides a Node.js HTTP service that generates TikTok API signatures (X-Bogus and X-Gnarly) required to access TikTok's data endpoints. It uses Puppeteer to run a headless Chromium browser, injects a bundled copy of TikTok's web SDK, and computes signatures by intercepting fetch requests. The service also includes a pure Node.js implementation for X-Gnarly using ChaCha20. It exposes POST /signature and POST /fetch endpoints, manages a request queue, and automatically refreshes the browser session after 500 signatures or 30 minutes. The repository includes example scripts for fetching user videos, comments, trending content, and search results, demonstrating typical TikTok data scraping workflows. While the core functionality is well-implemented, the reliance on static SDK copies and a heavy browser dependency introduces maintenance risks as TikTok updates its anti-scraping measures.
This repository is a GitHub Action designed to capture screenshots of websites across different operating systems. It takes inputs such as source URL or HTML content, destination filename, and optional parameters like full-page capture, width, and delay. The action uses the `capture-website` package (which relies on Puppeteer) to generate the screenshot, then uploads it as an artifact in the GitHub Actions workflow and exposes the file path as an output. The architecture is straightforward: the entry point (index.js) delegates to core logic in lib/screenshot-website.js, which handles input processing, Chrome executable location based on the runner OS, screenshot capture, artifact upload, and output setting. While the main logic is tightly coupled to the GitHub Actions environment, the repository contains a reusable utility function for normalizing and parsing action inputs (lib/load-inputs.js) that could be extracted for use in other Node.js projects. The action shows signs of technical drift due to its reliance on Node.js 12 (end-of-life) and outdated dependencies like Puppeteer 13.7.0, though it remains functional through active Dependabot updates and CI workflows.
SingleFileZ is a browser extension (Chrome/Firefox/Edge) and CLI tool that saves complete web pages as self-extracting HTML/ZIP polyglot files. The core output format embeds a ZIP archive containing all page resources (HTML, CSS, images, fonts, frames) inside an HTML wrapper with a JavaScript bootstrap that extracts and displays the page without external dependencies. The extension offers extensive configuration including multiple profiles, URL-based auto-save rules, cloud uploads (Google Drive, Dropbox, GitHub, WebDAV), an annotation editor for notes/highlights/cut operations, batch saving, and file name templating. Architecturally, the project is a multi-browser extension with background scripts orchestrating page capture, content scripts handling in-page UI and resource collection, and a task queue for managing parallel saves. It relies heavily on message passing between contexts and includes custom infrastructure: a binary serialization format (YABSON) for efficient data transfer, a full ZIP read/write library with support for encryption and streaming, and a fork of Mozilla's Readability.js for article extraction. The CLI component uses Docker to run headlessly, delegating to the `single-filez-cli` npm package. The codebase is large and feature-rich but shows signs of maintainability strain: core modules are distributed as minified build artifacts in `lib/`, the README states SingleFileZ has been merged into its parent project SingleFile (suggesting this repo is superseded), and it relies on Manifest v2 which is being phased out by browsers. The custom polyfills and browser-specific workarounds add complexity. Despite this, several components (YABSON serialization, the ZIP library, cloud upload abstractions) are well-isolated and could be reused independently with minimal modification.