Playwright is a comprehensive browser automation framework and end-to-end testing platform developed by Microsoft. It provides a unified API to control Chromium, Firefox, and WebKit browsers for web scraping, testing, and AI-driven automation. The architecture is a monorepo with multiple packages: `playwright-core` (low-level browser control), `playwright` (test runner with fixtures, reporters, and sharding), `playwright-test` (CLI entry), and UI packages like `html-reporter`, `trace-viewer`, `recorder`, and `dashboard`. Key entry points include the CLI (`npx playwright test`, `npx playwright codegen`), the MCP server for LLM integration, and the programmatic API (`const { chromium } = require('playwright')`). Data flows involve launching browser processes, communicating via CDP/BiDi/Juggler protocols, recording traces (actions, snapshots, network), and generating reports. External dependencies include Node.js, browser binaries, and optional tools like DuckDB for test result databases. The codebase demonstrates exceptional craftsmanship with clean separation of concerns, reusable isomorphic utilities, and a well-defined dispatcher pattern for client-server communication.
Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
browser-use is a comprehensive Python framework that enables AI agents to automate web browser interactions. It provides a unified interface for controlling browsers through Playwright/CDP, integrates with multiple LLM providers for decision-making, and includes sophisticated features like structured data extraction, security controls, token usage tracking, and sandboxed execution. The architecture is modular with clear separation between agent reasoning, browser management, tool execution, and extensibility mechanisms. Key innovations include a flexible action registry system with special parameter injection, domain-aware security watchdogs, automatic LLM cost tracking, and MCP integration for extending agent capabilities. The project demonstrates strong engineering practices with extensive testing, CI/CD pipelines, and comprehensive documentation through examples.
Invisible Dots is a self-hosted, open-source platform for deploying and managing persistent AI agents ('Dots'). Each Dot operates within its own dedicated QEMU virtual machine, providing a secure, isolated environment with a full Linux desktop, browser, and file system. The platform is designed for undetectability and long-running autonomy, serving as an open-source alternative to proprietary agent services. The system is a polyglot monorepo with a strict separation between the host control plane (TypeScript) and the guest agent (Go) and engine (Python). The host manages VM lifecycle, scheduling, a web UI, CLI, and messaging channels (Telegram/WhatsApp). The guest runs a Go daemon for low-level system access and a Python engine (forked from nanobot) that orchestrates LLM calls, tool execution, and permissions. Key features include a granular permission system with approval workflows, a cron-like task/automation system, cost capping per task, and a comprehensive benchmark suite for evaluating agent capabilities. The codebase demonstrates exceptional engineering craftsmanship with clean abstractions (Database, Channels, Computer), robust testing strategies (contract tests, fakes, e2e), and reusable utility libraries (ISO writer, SSE parser, atomic file operations). While primarily an application, the project includes a significant research-oriented benchmark suite (tests/bench) and the engine itself is a research fork (nanobot), giving it a hybrid character. The core value, however, is as a production-grade application platform.
Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
This repository is the official FastAPI full-stack template, providing a production-ready starter for full-stack web applications. It consists of a FastAPI backend (Python 3.14, SQLModel ORM, PostgreSQL, Alembic migrations) and a React frontend (TypeScript, Vite, TanStack Router/Query, Tailwind CSS, shadcn/ui). The template ships with complete authentication (JWT, password hashing with Argon2/Bcrypt, password recovery via email), user and item CRUD, role-based access control, and a responsive dashboard UI. Deployment is handled via Docker Compose with Traefik, and the project includes comprehensive CI/CD pipelines for testing (pytest, Playwright) and deployment (FastAPI Cloud or self-hosted). While it is a template, the code is structured for immediate extension and includes many reusable patterns such as generic CRUD utilities, email templates written in React Email, and auto-generated OpenAPI client code for the frontend. The repository is not a data research project; it is a pure application template with no data pipelines, notebooks, or ML components. It is actively maintained, with frequent commits and a strong community following.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
Windows Copilot API is an unofficial Python library and OpenAI-compatible HTTP server that bridges the consumer Microsoft Copilot chat service. It uses a pure-HTTP driver with curl_cffi to impersonate Chrome and speak Copilot's WebSocket protocol, handling authentication, proof-of-work challenges, and Cloudflare Turnstile clearance in-process. A Playwright-based browser module enables interactive sign-in and scheduled clearance refresh. The server wraps the library behind FastAPI endpoints exposing /v1/chat/completions and /v1/models with full streaming support, rate limiting, and concurrency serialization to cope with Copilot's single-account constraints. Notable features include transparent token refresh, conversation persistence, and a diagnostic tool for session debugging. The codebase is well-structured, with clear separation of concerns across protocol constants, challenge solvers, models, and drivers. It demonstrates advanced reverse-engineering and browser-emulation skills.
dembrandt is a production-grade design token extraction tool that analyzes live websites via headless Chromium (Playwright) and produces structured outputs including colors, typography, spacing, components, shadows, motion, and gradients. It supports multiple export formats: W3C DTCG tokens, Tailwind CSS v4 themes, DESIGN.md, self-contained HTML reports, PDF brand guides, and terminal display. Beyond extraction, it offers drift detection (comparing extractions over time with a 0–100 score), a GitHub Action for CI gates, and an MCP server for AI agent integration (Claude, Cursor). The architecture is modular with fault-isolated extractors, a pure-function drift engine, and a generic async job queue. While primarily an application, it includes an optional ONNX ML model for brand primary color prediction and advanced color science (OKLab, CIEDE2000, gamut mapping), giving it a hybrid character. The codebase demonstrates strong engineering practices: extensive test coverage, CI/CD pipelines, lazy browser loading, and graceful degradation across all output formatters.
Notte is a full-stack web AI agent framework and browser automation SDK. The implementation is a multi-package Python monorepo: notte-core provides shared domain models, actions, error hierarchies, telemetry, and configuration; notte-browser implements Playwright/Patchright browser control, DOM snapshotting, scraping pipelines, and form-filling heuristics; notte-agent contains the LLM-driven agent loop, deterministic fallback, and workflow execution; notte-llm wraps LiteLLM with structured completions, retries, and a file-based prompt library; notte-sdk is the client for the hosted Notte cloud API with remote sessions, agents, vaults, personas, and serverless functions; notte-integrations adds notifiers and third-party CDP session providers. Primary entry points are `NotteClient` for cloud usage, `NotteSession` for local browser automation, and `Agent`/`AgentFallback` for autonomous and hybrid scripted-agent workflows.
changedetection.io is a self-hosted web application for monitoring web pages for content changes. It periodically fetches URLs, computes diffs (text, JSON, image, or stock/price), and sends notifications via 100+ channels (Apprise). The system is built on Flask with Socket.IO for real-time UI updates, a pluggable processor architecture (text_json_diff, restock_diff, image_ssim_diff), and an LLM integration layer (via litellm) for intelligent change summarization and intent-based filtering. It supports multiple fetcher backends (requests, Playwright, Selenium, Puppeteer), browser automation steps, RSS feeds, and a REST API with OpenAPI spec. Architecturally, it is a monolith with background workers: a priority queue (RecheckPriorityQueue) dispatches watch rechecks to async workers, which fetch content, run processors, and trigger notifications. The datastore is file-based JSON with atomic writes and schema migrations. The frontend uses jQuery, SCSS, and custom JavaScript for diff visualization, visual selectors, and real-time queue monitoring. The plugin system (pluggy) allows extending fetchers, processors, conditions, and UI elements. Key entry points include the CLI (changedetection.py), the Flask app factory (flask_app.py), the worker pool (worker_pool.py), and the notification service (notification_service.py). External dependencies include Playwright, Apprise, litellm, lxml, and various image processing libraries (OpenCV, libvips). The codebase is well-tested with over 120 test files covering security, integration, and unit tests.
OpenCLI is a large-scale CLI framework that provides over 100 structured commands for interacting with diverse web services (social media, e-commerce, finance, academic databases, AI tools, etc.) via browser automation or direct HTTP APIs. It is designed for AI agents and developers who need deterministic, machine-readable access to websites. The architecture centers on a command registry pattern where each command declares metadata (site, name, access level, auth strategy, arguments, output columns) and either a function or a declarative pipeline of steps (navigate, evaluate, map, filter, sort, limit). Authentication is handled per-site via a shared helper that registers login/whoami commands. The tool manages browser sessions through a Chrome extension (Browser Bridge) and CDP, with a daemon for persistent connections. It includes a plugin system, shell completion, observation/tracing with redaction, and a skill system for AI agent documentation. The codebase is actively maintained with frequent releases, extensive unit tests, and e2e tests, but its heavy reliance on site-specific DOM selectors and API endpoints introduces fragility and maintenance burden.
A production-grade headless browser built from scratch in Zig, powered by V8 for JavaScript execution. It implements a vast subset of Web Platform APIs (DOM, CSSOM, HTML/SVG elements, canvas, WebGL, IndexedDB, workers, fetch, WebSocket, etc.) and exposes them through Chrome DevTools Protocol (CDP), WebDriver BiDi, and MCP (Model Context Protocol) servers. Includes a built-in LLM agent mode for natural-language-driven automation, a scripting system (PandaScript) for deterministic replay, and a full network stack with ad blocking, rate limiting, caching, and TLS. Designed for AI agents and automation workloads, it claims significant performance advantages over Chromium in startup time and resource usage. Architecturally, the browser is a monolithic server binary with distinct internal modules: a browser core managing pages, frames, and V8 isolates; a network layer using libcurl with connection pooling and request deduplication; protocol servers (CDP, BiDi, MCP) communicating over WebSocket or HTTP; and an agent subsystem that wraps LLM APIs for task planning and execution. The codebase is heavily tested with hundreds of hand-written HTML conformance tests and runs Web Platform Tests via CI. The engineering demonstrates high craftsmanship: custom arena allocators with V8 memory accounting, compile-time prototype hierarchy reflection for Web IDL bindings, a GCRA rate limiter, a singleflight pattern for request deduplication, and a generic live-collection iterator. While some APIs are stubbed or partially implemented (e.g., WebGL always returns null, video dimensions are hardcoded), the core functionality is extensive and spec-compliant where implemented. The project is actively maintained with nightly orderfile optimizations and regular CI.
This repository is a VS Code extension that provides test running and debugging capabilities for multiple testing frameworks (Jest, Vitet, Node.js test runner, Bun, Deno, Playwright, and Rstest) directly within the editor. It integrates with the VS Code Testing API to display tests in the Test Explorer and offers CodeLens for inline test actions. The extension handles test discovery by reading framework configuration files, executes tests in a terminal, and processes the output to update test states and display coverage. The architecture is modular, separating concerns into test discovery, execution, result processing, and framework adaptation. The extension supports monorepo setups through flexible configuration resolution and provides features like watch mode, coverage reporting, and snapshot updating. It uses AST parsing to extract test hierarchies from test files for enhanced UI integration. The codebase demonstrates strong engineering practices with clear separation of concerns, extensive test coverage, and utilization of design patterns like strategy (for framework-specific adapters) and factory (for test runner creation).
Playwright MCP is the npm distribution package for Microsoft's Model Context Protocol server that exposes Playwright browser automation to LLM clients as MCP tools. Despite the large star count, this repository contains almost no product source: the actual MCP tool implementation (browser_click, browser_navigate, browser_snapshot, network routing, storage, vision, PDF, devtools, and testing capabilities) lives in the Playwright monorepo at packages/playwright-core/src/tools/mcp. cli.js and index.js are thin shims that re-export tools.decorateMCPCommand and tools.createConnection from playwright-core's internal bundles, and src/README.md openly states where the source is. This repo functions as the release-shipping wrapper: it pins a Playwright alpha build, rolls it forward via roll.js, regenerates README tool/config documentation from the bundled tool schemas via update-readme.js, publishes the @playwright/mcp npm package (GitHub Actions OIDC, Azure ESRP, and the MCP Registry with server.json/package.json version validation), and builds multi-arch Docker images. Classified as a pure application, not data_research.
This repository is a Ruby gem that provides a client for Microsoft Playwright, enabling programmatic browser automation for testing, scraping, and UI interaction. It wraps the Playwright protocol by spawning a Node.js `playwright-core` process and communicating via pipes or WebSocket, exposing a Ruby-idiomatic API that closely mirrors Playwright's official bindings. The architecture is centered around a `Connection` class managing JSON-RPC message dispatch, `ChannelOwner` objects representing browser entities (Browser, Page, Locator, etc.), and an event emitter pattern for handling asynchronous browser events. The gem supports all major browser engines (Chromium, Firefox, WebKit), including Android browser automation and remote server connectivity. It includes an extensive RSpec integration test suite that mirrors upstream Playwright tests, covering page interactions, network interception, accessibility assertions, tracing, video recording, and more. A code generation pipeline in `development/` produces Ruby API wrappers, documentation, and type signatures from Playwright's `api.json`, ensuring alignment with each compatible Playwright version.
ScrapeGraphAI is a Python library that extracts structured data from websites and local documents using LLM-driven graph pipelines. Instead of a traditional web scraper with hardcoded selectors, users define extraction through natural-language prompts and optional Pydantic schemas; the library then executes a directed graph of nodes such as fetching, parsing, chunking, and answer generation. The public API is a set of graph classes (SmartScraperGraph, SearchGraph, JSONScraperGraph, CSVScraperGraph, SpeechGraph, ScriptCreatorGraph, etc.) in scrapegraphai/graphs, each composing reusable nodes from scrapegraphai/nodes and running them on a BaseGraph execution engine. It is consumed programmatically in Python and supports many LLM providers (OpenAI, Ollama, Groq, Mistral, Anthropic, DeepSeek, xAI, NVIDIA, MiniMax) and multiple fetching backends (Playwright, Selenium, Chromium, BrowserBase, Scrape.do). The architecture is a lightweight DAG framework: nodes have typed inputs/outputs, can branch conditionally, iterate, merge results, and accumulate cost/token metadata; BaseGraph walks the graph, handles state, and collects execution information. The repository also contains document loaders, model token/cost registries, prompt templates, telemetry, proxy rotation, screenshot-scraping utilities, a GraphBuilder that constructs graphs from natural language, and integrations such as Burr and Indexify. While the README aggressively promotes a commercial hosted API at scrapegraphai.com, the repository itself is a self-contained open-source library with examples and tests. The codebase is generally well structured and modular, with clear separation between graphs, nodes, prompts, helpers, utils, and models. It shows signs of rapid growth: some typos, duplicated test fixtures, hardcoded model pricing tables, a Python version mismatch between pyproject and Dockerfile, and dependency fragility around LangChain and optional browser/OCR packages. These issues are manageable but give the codebase a medium technical drift. Overall it is an active, useful library for LLM-based extraction, and several utilities are good candidates for reuse outside the project.
Artillery is a cloud-scale load testing and functional testing framework for backend services and APIs. It supports multiple protocols (HTTP, WebSocket, Socket.IO, Playwright, PostHog) through a plugin-based engine architecture. The system is primarily a CLI tool (built on oclif) that executes YAML/JSON test scripts defining virtual user flows with configurable arrival phases, payloads, and assertions. It provides distributed execution across local worker threads, AWS Lambda, AWS ECS/Fargate, and Azure ACI, with a bill-of-materials (BOM) system that bundles test dependencies for cloud workers. Metrics are aggregated via a custom Simple Statistics Measurement System (SSMS) supporting counters, histograms (DDSketch), and rates, and can be published to Datadog, CloudWatch, Prometheus, OpenTelemetry, Splunk, New Relic, and other observability platforms. The codebase is a monorepo managed with Turborepo, containing the core runner, platform integrations, first-party plugins (expect, ensure, apdex, slack, fake-data, metrics-by-endpoint, publish-metrics, memory-inspector), and shared type definitions. Key entry points include the `artillery` binary (`packages/artillery/bin/run`), the core runner (`packages/artillery/lib/core/runner.ts`), and platform launchers (`packages/artillery/lib/launch-platform.ts`). External dependencies include AWS SDK, Azure SDK, Playwright, got (HTTP client), and various metric publishing SDKs. The architecture is event-driven, using EventEmitter for lifecycle events and plugin hooks, with a global `global.artillery` object for shared state.
WaterCrawl is a full-stack web crawling and data extraction platform, built as a SaaS-style multi-tenant application. The backend is implemented in Django 5.1 with Django REST Framework, providing REST APIs for crawl requests (single/batch URL), web search (Google Custom Search integration), sitemap generation, and proxy server management. Asynchronous task execution is handled by Celery workers that invoke Scrapy spiders (SiteScrapper, GoogleCustomSearchScrapper, SitemapScrapper) and optionally use Playwright for JavaScript rendering. Real-time status updates are pushed to the frontend via Redis Pub/Sub and Server-Sent Events (SSE). The frontend is a React SPA (with TypeScript, Tailwind CSS, react-hook-form) that provides dashboard pages for configuring and monitoring crawl/search/sitemap jobs, managing teams and API keys, viewing usage statistics, and handling subscription billing through Stripe. Authentication supports JWT (simplejwt) and OAuth (Google, GitHub), with email verification and team invitation flows. The application stores extracted content (markdown, HTML, attachments) in MinIO or S3-compatible storage, and metadata in PostgreSQL.