This repository is a meticulously crafted template for using AI coding agents to reverse-engineer live websites into clean, modern Next.js codebases. It provides a pre-configured Next.js 16 scaffold (with shadcn/ui, Tailwind v4, and TypeScript strict) alongside a sophisticated set of agent instructions that guide the AI through the entire cloning process: mapping URLs, observing the target, building components, and comparing results. The project is not just a code scaffold; it is a complete workflow definition designed for Claude Code, Codex, Cursor, and OpenCode. The architecture is elegantly split between the application scaffold (src/, public/) and the agent skill definitions (.agents/skills/clone-website/). The canonical skill (SKILL.md) and its references (inspection-guide.md, framer-and-motion.md) represent a masterclass in prompt engineering, providing concrete, actionable guidance for handling complex scenarios like Framer-generated pages, scroll-driven animations, and responsive asset extraction. The project structure enforces clean separation of concerns for multiple cloned sites. The codebase demonstrates exceptional craftsmanship and a deep understanding of both software engineering and AI agent capabilities. The highlighted components—the agent skills, the token-counting CI workflow, and the cross-platform agent configuration—are genuinely reusable patterns that solve non-obvious problems. This is a rare example of a template that is more valuable for its workflow and documentation than its application code, setting a high standard for AI-assisted development projects. Maintenance is active, technical drift is low, and the project is a strong 9.0 on the quality scale.
Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
Cheerio is a fast, flexible, and elegant library for parsing and manipulating HTML and XML on the server side (Node.js). It provides a jQuery-like API for traversing and manipulating the DOM of parsed documents, designed for web scraping, data extraction, and HTML transformation tasks. The library abstracts away the underlying parser (htmlparser2 for speed, parse5 for standards compliance) via a factory pattern, offering a consistent Cheerio API regardless of parser choice. Key entry points include the main `load` function that converts HTML strings or buffers into a Cheerio instance, static methods like `html`, `xml`, `text`, and `parseHTML`, and utility functions for CSS handling, form serialization, and attribute manipulation. External dependencies include cheerio-select for CSS selector parsing, domutils for core DOM traversal, and optional HTTP fetching via undici for live URL parsing.
Scrapy is a mature, production-grade web scraping and crawling framework for Python. It provides a complete pipeline for downloading web pages, extracting structured data (via XPath/CSS selectors), and exporting it in multiple formats (JSON, CSV, XML, etc.). The architecture is event-driven, built on Twisted with full asyncio integration, and consists of an execution engine, downloader, scheduler, spider middleware, downloader middleware, item pipelines, and extensions. Key entry points include the CLI (`scrapy crawl`, `scrapy shell`, `scrapy startproject`), the `CrawlerProcess`/`CrawlerRunner` classes for programmatic use, and the `Spider` base class for user-defined spiders. Data flows from HTTP requests through the downloader (with support for HTTP/1.1, HTTP/2, FTP, and experimental httpx), through spider callbacks, into item pipelines, and finally to feed exporters or custom storage backends. External dependencies include Twisted, lxml, parsel, w3lib, protego (robots.txt), queuelib, and optional integrations like boto3 (S3), google-cloud-storage, Pillow (image processing), and mitmproxy (testing). The framework is highly configurable via a priority-based settings system and supports plugins (addons, middlewares, pipelines, extensions).
SearXNG is a privacy-respecting metasearch engine that aggregates results from over 100 search engines (Google, Bing, DuckDuckGo, Wikipedia, academic sources, etc.) via HTTP requests. It runs as a Flask web application, supports multiple output formats (HTML, JSON, RSS), and includes a plugin system, bot detection, rate limiting, and user preferences stored in cookies. The architecture is a monolithic Python application with a separate TypeScript/Less frontend built with Vite. It uses Valkey/Redis for caching and rate limiting, and SQLite for persistent caches. The codebase is well-structured with a clear separation of concerns: engine modules implement a common request/response interface, result types are typed with msgspec, and plugins are registered via a simple hook system. It is actively maintained with CI/CD, Docker support, and comprehensive documentation.
Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
Etaoin is a pure Clojure library that implements the W3C WebDriver protocol for automating major browsers (Chrome, Firefox, Safari, Edge). It provides a comprehensive, idiomatic Clojure API for browser control—navigation, element interaction, JavaScript execution, screenshots, shadow DOM, and more—without requiring Selenium or Java interop. The library also includes a Selenium IDE (.side) file executor, Chrome DevTools performance log parsing, and a rich macro DSL for resource management (with-driver, with-wait, with-postmortem). Internally, it uses multimethod dispatch to handle browser-specific quirks, a generic polling mechanism for waits, and a composable action builder for complex input sequences. The codebase is well-structured, extensively tested across multiple browsers and runtimes (JVM Clojure and Babashka), and provides clj-kondo hooks for linting its macros. It is primarily a library for integration into other projects (UI testing, web scraping, automation), but also offers a CLI entry point for running IDE scripts.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
Mechanize is a mature Ruby library for automating web interactions. It acts as a programmable web browser, enabling users to fetch pages, follow links, submit forms, manage cookies, handle authentication (Basic, Digest, NTLM), and support content compression (gzip, deflate, brotli, zstd). The library wraps `net-http-persistent` for connection management and `nokogiri` for HTML/XML parsing. Key architectural patterns include a pluggable parser system that maps MIME types to response handlers (e.g., `Mechanize::Page` for HTML, `Mechanize::Image`, `Mechanize::Download`), an element matching mixin for filtering links and forms, a robust cookie jar, and an authentication store. Its test suite (over 150 test files) uses a WEBrick-based mock server and servlets to simulate realistic scenarios without network calls. The library is widely used for scraping, testing, and automation in Ruby ecosystems.
ProxyPool is a production-grade proxy pool application that automatically collects free HTTP/HTTPS proxies from over a dozen public sources, validates their availability, and stores them in Redis or SSDB. It provides a Flask RESTful API for consuming proxies (get, pop, all, count, delete) and uses APScheduler for periodic fetching and validation. The architecture is a monolith with two main processes: a scheduler that runs fetch/check cycles and an API server (Flask with Gunicorn on Linux). The codebase demonstrates strong software engineering practices: a plugin-based fetcher system with dynamic module loading and caching, a clean Proxy data model with JSON serialization, a decorator-based validator registration pattern, and a layered test suite (unit, API, integration) with fakeredis mocking. External dependencies include Redis/SSDB, APScheduler, Flask, requests, and lxml. The project is actively maintained with recent commits and a comprehensive test suite.
Defuddle is a content extraction library that converts web pages into clean, readable Markdown with structured metadata (title, author, site, published date). It is the engine behind the Obsidian Web Clipper browser extension and also runs as a standalone CLI tool, an npm package, and a hosted API service on Cloudflare Workers. The architecture follows a modular pipeline: fetch HTML, detect site-specific extractors, score and select main content, remove clutter (nav, ads, sidebars), standardize complex elements (footnotes, math, code blocks, images), and convert to Markdown via Turndown. The library is designed as a more forgiving alternative to Mozilla's Readability, with support for mobile-style heuristics, schema.org metadata extraction, and extensive site-specific extractors for platforms like YouTube, Reddit, Twitter/X, ChatGPT, GitHub, and Gmail. The codebase is written in TypeScript and produces three bundles: core (browser, no extra dependencies), full (adds MathML-to-LaTeX conversion), and node (accepts any DOM implementation). The CLI uses Commander and supports file, URL, and stdin input with options for Markdown, JSON, frontmatter, and debug output. The hosted API runs on Cloudflare Workers with Stripe-based billing, rate limiting, and Durable Objects for stateful API key management. The test suite is exceptionally comprehensive, with hundreds of HTML fixtures and expected Markdown outputs covering edge cases from dozens of real-world websites, CMS patterns, and syntax highlighters. Key external dependencies include Turndown (Markdown conversion), linkedom (DOM parsing in Node.js), mathml-to-latex and Temml (math rendering), and various third-party APIs for site-specific extractors (YouTube InnerTube, FxTwitter, Reddit Atom feeds, Bilibili). The project maintains strict security practices: no innerHTML assignments outside a single utility function, URL sanitization, and XSS prevention through template-based HTML parsing. The build system uses webpack with aliases for math module switching and enforces bundle size budgets via CI scripts.
MechanicalSoup is a Python library for automating interaction with websites. It simulates a web browser by automatically storing and sending cookies, following redirects, and submitting forms. It does not execute JavaScript, making it a lightweight alternative to Selenium for sites that rely on server-side logic. The library is built on top of Requests for HTTP sessions and BeautifulSoup for HTML parsing and manipulation. The architecture provides two main classes: Browser (low-level, stateless) and StatefulBrowser (high-level, stateful). StatefulBrowser is the recommended interface, maintaining the current page, URL, and selected form as internal state. Users can open URLs, follow links, select forms, fill in fields, and submit. The Form class handles form manipulation including text inputs, checkboxes, radio buttons, select dropdowns, textareas, and file uploads. It also correctly implements HTML form submission semantics such as disabled controls, fieldset disabling, and multipart encoding. Key entry points include StatefulBrowser.open(), select_form(), submit_selected(), follow_link(), and download_link(). The library depends on requests, beautifulsoup4, and lxml. It is designed as a reusable library, not a standalone application, and is typically installed via pip and imported into user scripts. The codebase is well-tested with 100% coverage, has comprehensive documentation, and follows semantic versioning.
Huginn is a self-hosted, hackable automation platform built as a Ruby on Rails application. It allows users to create and chain 'agents' that monitor, scrape, transform, and act on data from hundreds of sources (web, RSS, email, Twitter, Telegram, Slack, OpenAI, etc.) via an event-driven architecture. Agents are polymorphic Ruby classes with a consistent lifecycle: scheduled checks, event reception, and event emission. The system supports Liquid templating for dynamic configuration, OAuth services, file pointers, and background job processing via DelayedJob. The codebase is extensive with 80+ agent types, reusable concerns (WebRequestConcern, LiquidInterpolatable, FormConfigurable), and a comprehensive test suite. While primarily an application, it includes a small research dataset (ANEW sentiment norms) used by a sentiment analysis agent, but the dominant character is clearly a production automation service.
GPT Researcher is an autonomous research agent that produces comprehensive, cited reports from web and local sources. In production, it operates as a full-stack web application: a FastAPI backend (main.py, backend/server/app.py) exposes REST and WebSocket endpoints for initiating research, streaming progress, and retrieving reports. A Next.js frontend (frontend/nextjs/) provides the user interface, and a lightweight vanilla JS frontend (frontend/scripts.js) is also available. The core Python package (gpt_researcher/) can be used programmatically via the GPTResearcher class or as a CLI tool (cli.py). Architecture follows a planner-executor pattern: given a research task, the system generates sub-queries, executes them in parallel across 20+ search engine wrappers (Tavily, Google, Bing, Brave, DuckDuckGo, ArXiv, PubMed, etc.), scrapes the resulting URLs using multiple backends (BeautifulSoup, browser automation, PyMuPDF, FireCrawl), filters and compresses the context using embedding or keyword-based methods, and finally generates a structured report with citations. The system supports multiple report types (summary, detailed, deep research) and multi-agent orchestrations (LangGraph, AG2) for more complex workflows. Key data flows: user query to sub-query generation (LLM) to parallel search across configured retrievers to URL scraping to context compression to source curation to report generation (LLM) to output as markdown/PDF/DOCX. Real-time progress is streamed via WebSocket. External dependencies include LLM providers (OpenAI, Anthropic, Google, Ollama, etc.), search APIs (Tavily, SerpAPI, etc.), and optional services (MCP servers, vector stores, image generation APIs). The codebase demonstrates strong engineering practices: extensive test suite (200+ tests), plugin architecture for retrievers, configurable context filters, cost tracking, and evaluation frameworks for quality and hallucination. However, it carries maintenance risks from its large dependency surface, multiple parallel agent implementations, and reliance on rapidly-changing external APIs.
This is a Deno/TypeScript library for extracting structured article data (title, content, author, date, image, etc.) from any URL or HTML string. It uses Mozilla's Readability for the core extraction, supplemented by custom metadata extraction from HTML meta tags, JSON-LD schemas, and link elements. The library provides a transformation system that allows per-site pre/post processing of HTML to improve extraction quality, and a flexible HTML sanitizer to control output tags and attributes. The architecture is modular, with clear separation of concerns: retrieval, metadata extraction, URL normalization, readability extraction, sanitization, and transformation. Entry points are `extract()` and `extractFromHtml()`, with optional custom fetch functions and parser options. The codebase is well-tested, actively maintained, and demonstrates solid engineering practices with minimal dependencies.
omni-article-markdown is a CLI tool and library that converts web articles from various sources into clean Markdown format. It employs a modular pipeline architecture: Reader fetches HTML (using requests or Playwright for JavaScript-heavy sites), Extractor isolates article content by removing boilerplate and ads, and Parser converts the cleaned HTML to Markdown with support for code blocks, tables, math equations, and GitHub Gists. The system uses a plugin-based architecture where site-specific readers and extractors are automatically discovered and loaded, allowing easy extension for new websites. Key features include SSL verification control, cookie persistence for sites requiring authentication (like Zhihu), and flexible output options (stdout or file saving).
changedetection.io is a self-hosted web application for monitoring web pages for content changes. It periodically fetches URLs, computes diffs (text, JSON, image, or stock/price), and sends notifications via 100+ channels (Apprise). The system is built on Flask with Socket.IO for real-time UI updates, a pluggable processor architecture (text_json_diff, restock_diff, image_ssim_diff), and an LLM integration layer (via litellm) for intelligent change summarization and intent-based filtering. It supports multiple fetcher backends (requests, Playwright, Selenium, Puppeteer), browser automation steps, RSS feeds, and a REST API with OpenAPI spec. Architecturally, it is a monolith with background workers: a priority queue (RecheckPriorityQueue) dispatches watch rechecks to async workers, which fetch content, run processors, and trigger notifications. The datastore is file-based JSON with atomic writes and schema migrations. The frontend uses jQuery, SCSS, and custom JavaScript for diff visualization, visual selectors, and real-time queue monitoring. The plugin system (pluggy) allows extending fetchers, processors, conditions, and UI elements. Key entry points include the CLI (changedetection.py), the Flask app factory (flask_app.py), the worker pool (worker_pool.py), and the notification service (notification_service.py). External dependencies include Playwright, Apprise, litellm, lxml, and various image processing libraries (OpenCV, libvips). The codebase is well-tested with over 120 test files covering security, integration, and unit tests.
Kazumi is a cross-platform Flutter application for browsing, collecting, and streaming anime (番剧). At its core, it provides a curated media experience: users can discover anime via popularity rankings, weekly schedules, text and image (trace.moe) search; manage a local watchlist synced with Bangumi (bgm.tv) and WebDAV; and play videos using a hardware-accelerated media_kit (libmpv) player with danmaku overlay, Anime4K super-resolution shaders, Picture-in-Picture, and external player support. It also downloads episodes (HLS segments) for offline playback and supports synchronized group viewing via the Syncplay protocol. The application is built as a Flutter monolith with modular routing (flutter_modular), reactive state management (MobX), local storage (Hive), and a custom plugin system that defines scraping rules (XPath/API) to extract video sources from third-party websites. External dependencies include Bangumi, DanDanPlay, trace.moe, WebDAV servers, and a GitHub-hosted plugin catalog. The codebase demonstrates mature engineering practices including well-isolated async utilities, event-sourced history sync with conflict resolution, and platform-specific WebView integrations for video source extraction and captcha handling.
OpenCLI is a large-scale CLI framework that provides over 100 structured commands for interacting with diverse web services (social media, e-commerce, finance, academic databases, AI tools, etc.) via browser automation or direct HTTP APIs. It is designed for AI agents and developers who need deterministic, machine-readable access to websites. The architecture centers on a command registry pattern where each command declares metadata (site, name, access level, auth strategy, arguments, output columns) and either a function or a declarative pipeline of steps (navigate, evaluate, map, filter, sort, limit). Authentication is handled per-site via a shared helper that registers login/whoami commands. The tool manages browser sessions through a Chrome extension (Browser Bridge) and CDP, with a daemon for persistent connections. It includes a plugin system, shell completion, observation/tracing with redaction, and a skill system for AI agent documentation. The codebase is actively maintained with frequent releases, extensive unit tests, and e2e tests, but its heavy reliance on site-specific DOM selectors and API endpoints introduces fragility and maintenance burden.