Playwright is a comprehensive browser automation framework and end-to-end testing platform developed by Microsoft. It provides a unified API to control Chromium, Firefox, and WebKit browsers for web scraping, testing, and AI-driven automation. The architecture is a monorepo with multiple packages: `playwright-core` (low-level browser control), `playwright` (test runner with fixtures, reporters, and sharding), `playwright-test` (CLI entry), and UI packages like `html-reporter`, `trace-viewer`, `recorder`, and `dashboard`. Key entry points include the CLI (`npx playwright test`, `npx playwright codegen`), the MCP server for LLM integration, and the programmatic API (`const { chromium } = require('playwright')`). Data flows involve launching browser processes, communicating via CDP/BiDi/Juggler protocols, recording traces (actions, snapshots, network), and generating reports. External dependencies include Node.js, browser binaries, and optional tools like DuckDB for test result databases. The codebase demonstrates exceptional craftsmanship with clean separation of concerns, reusable isomorphic utilities, and a well-defined dispatcher pattern for client-server communication.
browser-use is a comprehensive Python framework that enables AI agents to automate web browser interactions. It provides a unified interface for controlling browsers through Playwright/CDP, integrates with multiple LLM providers for decision-making, and includes sophisticated features like structured data extraction, security controls, token usage tracking, and sandboxed execution. The architecture is modular with clear separation between agent reasoning, browser management, tool execution, and extensibility mechanisms. Key innovations include a flexible action registry system with special parameter injection, domain-aware security watchdogs, automatic LLM cost tracking, and MCP integration for extending agent capabilities. The project demonstrates strong engineering practices with extensive testing, CI/CD pipelines, and comprehensive documentation through examples.
Puppeteer is a high-level Node.js library for browser automation, primarily targeting Chrome/Chromium and Firefox. It provides a rich API to control browsers programmatically via the Chrome DevTools Protocol (CDP) and WebDriver BiDi, supporting both headless and headed modes. The library is organized as a monorepo with multiple packages: `puppeteer` (full library with browser management), `puppeteer-core` (core automation API without browser download), `@puppeteer/browsers` (browser download and CLI), `@puppeteer/ng-schematics` (Angular integration), and a test server. The architecture features abstract API classes (Page, Browser, ElementHandle) with separate CDP and BiDi implementations, a custom event system, and extensive use of TypeScript generics and resource management patterns. Puppeteer is widely used for end-to-end testing, web scraping, and automation tasks, with a strong emphasis on cross-browser compatibility and modern web platform features.
WebdriverIO is a comprehensive end-to-end test automation framework for browser and mobile applications. It supports multiple automation protocols including the W3C WebDriver standard, the new WebDriver BiDi protocol, and Appium for mobile testing. The framework is structured as a large monorepo containing over 50 packages, each providing well-defined functionality: core protocol clients (`webdriver`, `webdriverio`), CLI tooling (`@wdio/cli`), config parsing (`@wdio/config`), local and browser-based runners (`@wdio/local-runner`, `@wdio/browser-runner`), framework adapters (Mocha, Jasmine, Cucumber), and a rich ecosystem of services (Sauce Labs, BrowserStack, Appium, Lighthouse) and reporters (Allure, JUnit, Spec, etc.). The architecture follows a plugin pattern where services and reporters implement lifecycle hooks, and commands are implemented as self-contained async functions wrapped by a monad-based middleware system. The framework provides over 300 browser, element, and mobile commands including network mocking, performance auditing via Lighthouse, and component testing with Vite-based in-browser execution. Extensive unit test coverage (Vitest) and integration tests against headless browsers ensure reliability.
Jint is a high-performance JavaScript interpreter for .NET, implementing the full ECMAScript specification (ES2025+) with extensive Web API support (Fetch, Streams, Crypto, Encoding, Workers, etc.) and a headless browser automation layer. The core engine provides a rich embedding API with CLR interop, module loading, execution constraints, and debugging via the Chrome DevTools Protocol. The architecture is modular: the engine itself is a library (Jint), with separate packages for DevTools debugging (Jint.DevTools), browser automation (Jint.Browser), and source generators for efficient interop (Jint.SourceGenerators). The codebase is exceptionally well-tested with thousands of unit tests, Web Platform Tests (WPT), and Test262 conformance tests, plus a comprehensive benchmark suite. It is used in production by projects like RavenDB, EventStore, and OrchardCore as an embedded scripting engine.
agent-browser is a native Rust CLI tool designed for browser automation specifically tailored for AI agent workflows. It employs a client-daemon architecture where the Rust daemon manages persistent browser sessions via Chrome DevTools Protocol (CDP) or WebDriver, while the CLI provides command-line interface for controlling browsers. The system includes a Next.js dashboard for manual session monitoring and control, an MCP server for standardized AI agent tool integration, and a skills system for sharing and discovering agent workflows. Key features include snapshot-ref-act workflows for interactive element manipulation, secure credential vaults, provider integrations (Browserbase, Browserless, etc.), AI chat panels via Vercel AI Gateway, and extensive observability capabilities. The project emphasizes security through content boundaries, domain allowlists, and action policies, making it suitable for production AI agent deployments.
Invisible Dots is a self-hosted, open-source platform for deploying and managing persistent AI agents ('Dots'). Each Dot operates within its own dedicated QEMU virtual machine, providing a secure, isolated environment with a full Linux desktop, browser, and file system. The platform is designed for undetectability and long-running autonomy, serving as an open-source alternative to proprietary agent services. The system is a polyglot monorepo with a strict separation between the host control plane (TypeScript) and the guest agent (Go) and engine (Python). The host manages VM lifecycle, scheduling, a web UI, CLI, and messaging channels (Telegram/WhatsApp). The guest runs a Go daemon for low-level system access and a Python engine (forked from nanobot) that orchestrates LLM calls, tool execution, and permissions. Key features include a granular permission system with approval workflows, a cron-like task/automation system, cost capping per task, and a comprehensive benchmark suite for evaluating agent capabilities. The codebase demonstrates exceptional engineering craftsmanship with clean abstractions (Database, Channels, Computer), robust testing strategies (contract tests, fakes, e2e), and reusable utility libraries (ISO writer, SSE parser, atomic file operations). While primarily an application, the project includes a significant research-oriented benchmark suite (tests/bench) and the engine itself is a research fork (nanobot), giving it a hybrid character. The core value, however, is as a production-grade application platform.
Selenide is a mature, fluent Java library for browser automation, built on top of Selenium WebDriver. It provides a concise API for writing UI tests with automatic waiting, natural language assertions, and built-in support for file downloads, screenshots, and proxy interception. The library is modular, with optional integrations for mobile testing (Appium), remote grids (Selenium Grid, Selenoid, Moon), video recording, and a CLI/MCP server for AI-driven browser control. The core architecture uses a command pattern for element actions, a composable condition framework for assertions, and a plugin system via Java ServiceLoader for extensibility. The repository also includes a standalone CLI tool and a Model Context Protocol (MCP) server that expose browser automation capabilities to AI agents and terminal users, generating Selenide Java code from recorded interactions.
Gstack is a production-grade AI agent skill framework and headless browser automation toolchain designed for Claude Code, Codex, Cursor, and other coding agents. It provides a structured skill system (50+ slash-command skills like `/ship`, `/review`, `/plan-ceo-review`) with a resolver pipeline that generates per-host `SKILL.md` templates from TypeScript sources. The repo also contains a fully functional headless browser daemon (`browse/`) with anti-bot stealth, multi-layer prompt injection defense (canary tokens, ML classifiers, URL blocklist, envelope sentinel), CDP escape hatch, SOCKS5 proxy bridge, and secure file I/O. Additional subsystems include markdown-to-PDF generation (`make-pdf/`), iOS QA automation (`ios-qa/`), design mockup generation (`design/`), a secret redaction engine, and a cross-session knowledge base (gbrain). The codebase is unusually well-tested with extensive unit, integration, and periodic paid E2E tests against real LLM APIs. While primarily an application, it includes data pipeline components like telemetry aggregation, eval result stores, and a brain cache specification with TTL-based entities and calibration weights, making it a hybrid with strong data engineering discipline.
Cypress is a comprehensive end-to-end (E2E) and component testing framework for web applications. It operates as a monorepo containing a CLI (cli), a Node.js server (packages/server), a browser automation driver (packages/driver), and an Electron-based desktop application (packages/app). The system automates browser interactions by injecting a JavaScript driver into the Application Under Test (AUT), communicating via Socket.io and a custom HTTP proxy for network interception. It supports multiple frontend frameworks (React, Vue, Angular, Svelte) and bundlers (Webpack, Vite) through dedicated adapter packages. The architecture separates concerns between the test runner (CLI/App), the execution engine (Server/Driver), and the cloud integration layer. Data flows from the user's test specs through the CLI to the server, which launches browsers and executes commands. Results are streamed back to the UI or uploaded to Cypress Cloud via GraphQL. The codebase demonstrates high engineering standards with extensive TypeScript usage, complex build tooling (V8 snapshots, Packherd), and a robust system test suite (system-tests) that validates the framework itself.
Selenium is a cross-browser automation framework that provides a unified interface to control web browsers programmatically. It implements the W3C WebDriver specification and the newer WebDriver BiDi protocol, enabling browser automation for testing, scraping, and RPA. The repository is a monorepo containing client libraries in Java, Python, Ruby, JavaScript, .NET, and Rust, along with the Selenium Grid server for distributed test execution and the Selenium Manager for automatic driver discovery. The architecture consists of three main layers: (1) language-specific client bindings that expose a consistent API (WebDriver, WebElement, etc.) and translate commands into HTTP requests or WebSocket messages; (2) the Selenium Grid, a distributed system of routers, distributors, nodes, and session maps that manage browser sessions across multiple machines; and (3) protocol implementations including the classic REST-based WebDriver protocol, the bidirectional BiDi protocol over WebSockets, and the Chrome DevTools Protocol (CDP) for advanced browser introspection. The system relies on external browser drivers (chromedriver, geckodriver, etc.) and communicates with browsers via HTTP and WebSocket. Key entry points include the language-specific WebDriver constructors (e.g., webdriver.Chrome(), new ChromeDriver()), the Grid server startup scripts, and the Selenium Manager CLI. Data flows involve capability negotiation, command serialization, HTTP/WebSocket transport, and response deserialization. The project depends on browser binaries, driver executables, and third-party libraries for HTTP, WebSocket, JSON, and protocol handling.
Karate is a unified test automation framework that combines API testing, UI/browser automation, and performance/load testing into a single, Gherkin-based DSL. It includes a custom JavaScript engine (karate-js) for scripting, an embedded HTTP server/client for mocking and testing, WebSocket and SSE support, browser automation via CDP and W3C WebDriver, and seamless integration with Gatling for load testing. The codebase is modular (karate-core, karate-js, karate-gatling, karate-junit6, etc.) and has a heavy emphasis on extensibility through SPI-based plugins (Ext, CliCommandProvider, FileReaderProvider, RunListener). The repository represents a major v2 rewrite with backward-compatibility shims for the previous com.intuit.karate package. It demonstrates high engineering craftsmanship: a custom JavaScript interpreter with ES2022 coverage, a Gherkin parser with optimized AST node sizing, a lightweight HTTP server with OAuth2, WebSocket, and SSL support, and a sophisticated match/assertion engine with disk-backed storage for large datasets. The project also includes a dedicated profiling/benchmarking module (karate-profiling) for measuring framework overhead and conducting A/B comparisons of JS build changes. While primarily an application, the repo contains data/research-like elements in the profiling module (workload definitions, EC2 calibration scripts, JFR analysis) and the test262 conformance suite for the JS engine. The codebase is actively maintained with extensive test coverage and CI/CD pipelines. Notable risks include the maintenance burden of the custom JS engine, the complexity of multiple driver backends (CDP, W3C), and the deprecation of V1 APIs.
Etaoin is a pure Clojure library that implements the W3C WebDriver protocol for automating major browsers (Chrome, Firefox, Safari, Edge). It provides a comprehensive, idiomatic Clojure API for browser control—navigation, element interaction, JavaScript execution, screenshots, shadow DOM, and more—without requiring Selenium or Java interop. The library also includes a Selenium IDE (.side) file executor, Chrome DevTools performance log parsing, and a rich macro DSL for resource management (with-driver, with-wait, with-postmortem). Internally, it uses multimethod dispatch to handle browser-specific quirks, a generic polling mechanism for waits, and a composable action builder for complex input sequences. The codebase is well-structured, extensively tested across multiple browsers and runtimes (JVM Clojure and Babashka), and provides clj-kondo hooks for linting its macros. It is primarily a library for integration into other projects (UI testing, web scraping, automation), but also offers a CLI entry point for running IDE scripts.
This repository is a Bilibili automation tool designed to automatically claim live stream rewards (drops) and complete watching time tasks. It features a dual-mode interface with both a PySide6-based GUI and a CLI entry point. The core functionality involves browser automation via Selenium to extract necessary credentials (cookies) and task information from Bilibili's web interface, followed by API interactions using signed requests (WBI) to claim rewards and report watch time via x25Kn heartbeat protocols. The application includes a notification system supporting multiple services (Gotify, Server酱, WeCom, Apprise) and persists configuration via JSON files and Qt settings. The architecture is modular, separating concerns into distinct modules for API client logic, worker management, GUI components, and utilities.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
Notte is a full-stack web AI agent framework and browser automation SDK. The implementation is a multi-package Python monorepo: notte-core provides shared domain models, actions, error hierarchies, telemetry, and configuration; notte-browser implements Playwright/Patchright browser control, DOM snapshotting, scraping pipelines, and form-filling heuristics; notte-agent contains the LLM-driven agent loop, deterministic fallback, and workflow execution; notte-llm wraps LiteLLM with structured completions, retries, and a file-based prompt library; notte-sdk is the client for the hosted Notte cloud API with remote sessions, agents, vaults, personas, and serverless functions; notte-integrations adds notifiers and third-party CDP session providers. Primary entry points are `NotteClient` for cloud usage, `NotteSession` for local browser automation, and `Agent`/`AgentFallback` for autonomous and hybrid scripted-agent workflows.
gpt4free (g4f) is a large-scale, community-driven Python library that aggregates dozens of free AI providers (LLMs, image, audio, video) into a unified, OpenAI-compatible interface. It provides multiple access modes: a Python client library (sync/async), a FastAPI-based API server, a Flask web GUI, a CLI tool, a Model Context Protocol (MCP) server for tool calling, a browser extension, a Discord bot, an Android app, and a Go launcher. The architecture is provider-based: each AI service (e.g., ChatGPT, Gemini, DeepSeek, Copilot, Grok, Perplexity) is wrapped in an adapter class implementing a common `AsyncGeneratorProvider` interface. The system handles authentication via cookies, OAuth flows, HAR files, and browser automation (CDP/nodriver), and supports streaming, tool/function calling, image generation, audio synthesis, and web search. It is primarily an application for consuming free AI models, not a research or data pipeline project, though it includes some data-oriented utilities like token optimization and usage tracking.
This repository implements an MCP (Model Context Protocol) server that exposes Chrome DevTools capabilities as tools for AI coding assistants. In production, it launches a headless or headed Chrome browser via Puppeteer, connects to the DevTools frontend SDK, and provides over 50 tools covering navigation, input automation, network inspection, performance tracing, heap snapshot analysis, CSS inspection, accessibility auditing (via Lighthouse), PWA management, and extension debugging. The architecture is a Node.js server that runs either as a long-lived daemon (Unix socket IPC) or as a CLI tool, with a configuration pipeline supporting CLI flags, config files, and environment variables. Key entry points are `src/bin/chrome-devtools-mcp.ts` (MCP server) and `src/bin/chrome-devtools.ts` (standalone CLI). Data flows involve collecting console messages, network requests, and DevTools issues per page via `PageCollector`, processing performance traces via `ChunkedTraceParser` and `PerformanceTrace`, and formatting results through specialized formatters (`ConsoleFormatter`, `CssFormatter`, `HeapSnapshotFormatter`, etc.) that support both text and structured JSON output. External dependencies include Puppeteer for browser control, the DevTools frontend SDK for advanced features (heap snapshots, stack traces), Lighthouse for audits, and a telemetry system (Clearcut) for usage statistics.
OpenCLI is a large-scale CLI framework that provides over 100 structured commands for interacting with diverse web services (social media, e-commerce, finance, academic databases, AI tools, etc.) via browser automation or direct HTTP APIs. It is designed for AI agents and developers who need deterministic, machine-readable access to websites. The architecture centers on a command registry pattern where each command declares metadata (site, name, access level, auth strategy, arguments, output columns) and either a function or a declarative pipeline of steps (navigate, evaluate, map, filter, sort, limit). Authentication is handled per-site via a shared helper that registers login/whoami commands. The tool manages browser sessions through a Chrome extension (Browser Bridge) and CDP, with a daemon for persistent connections. It includes a plugin system, shell completion, observation/tracing with redaction, and a skill system for AI agent documentation. The codebase is actively maintained with frequent releases, extensive unit tests, and e2e tests, but its heavy reliance on site-specific DOM selectors and API endpoints introduces fragility and maintenance burden.
This repository implements a Cloudflare Worker that runs the OpenClaw personal AI assistant inside a Cloudflare Sandbox container. The worker acts as a reverse proxy, forwarding HTTP and WebSocket traffic to the OpenClaw gateway running on port 18789 inside the container. It provides an admin UI (React SPA) for device pairing management, REST API endpoints for device approval and storage sync, a Chrome DevTools Protocol (CDP) shim for browser automation via Cloudflare Browser Rendering, and debug routes for troubleshooting. The architecture is a single Hono application with middleware for logging, environment validation, and Cloudflare Access authentication. The catch-all route proxies requests to the container, with special handling for WebSocket connections (message interception for error transformation) and crash recovery (automatic gateway restart with R2 backup restore). The codebase is well-structured with TypeScript strict mode, comprehensive unit tests, and a full end-to-end test suite that deploys to real Cloudflare infrastructure. Notable features include a robust process lifecycle manager for the gateway, a cron-based container wake mechanism, and a CDP shim that translates CDP commands to Puppeteer calls.