Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
Huginn is a self-hosted, hackable automation platform built as a Ruby on Rails application. It allows users to create and chain 'agents' that monitor, scrape, transform, and act on data from hundreds of sources (web, RSS, email, Twitter, Telegram, Slack, OpenAI, etc.) via an event-driven architecture. Agents are polymorphic Ruby classes with a consistent lifecycle: scheduled checks, event reception, and event emission. The system supports Liquid templating for dynamic configuration, OAuth services, file pointers, and background job processing via DelayedJob. The codebase is extensive with 80+ agent types, reusable concerns (WebRequestConcern, LiquidInterpolatable, FormConfigurable), and a comprehensive test suite. While primarily an application, it includes a small research dataset (ANEW sentiment norms) used by a sentiment analysis agent, but the dominant character is clearly a production automation service.
GPT Researcher is an autonomous research agent that produces comprehensive, cited reports from web and local sources. In production, it operates as a full-stack web application: a FastAPI backend (main.py, backend/server/app.py) exposes REST and WebSocket endpoints for initiating research, streaming progress, and retrieving reports. A Next.js frontend (frontend/nextjs/) provides the user interface, and a lightweight vanilla JS frontend (frontend/scripts.js) is also available. The core Python package (gpt_researcher/) can be used programmatically via the GPTResearcher class or as a CLI tool (cli.py). Architecture follows a planner-executor pattern: given a research task, the system generates sub-queries, executes them in parallel across 20+ search engine wrappers (Tavily, Google, Bing, Brave, DuckDuckGo, ArXiv, PubMed, etc.), scrapes the resulting URLs using multiple backends (BeautifulSoup, browser automation, PyMuPDF, FireCrawl), filters and compresses the context using embedding or keyword-based methods, and finally generates a structured report with citations. The system supports multiple report types (summary, detailed, deep research) and multi-agent orchestrations (LangGraph, AG2) for more complex workflows. Key data flows: user query to sub-query generation (LLM) to parallel search across configured retrievers to URL scraping to context compression to source curation to report generation (LLM) to output as markdown/PDF/DOCX. Real-time progress is streamed via WebSocket. External dependencies include LLM providers (OpenAI, Anthropic, Google, Ollama, etc.), search APIs (Tavily, SerpAPI, etc.), and optional services (MCP servers, vector stores, image generation APIs). The codebase demonstrates strong engineering practices: extensive test suite (200+ tests), plugin architecture for retrievers, configurable context filters, cost tracking, and evaluation frameworks for quality and hallucination. However, it carries maintenance risks from its large dependency surface, multiple parallel agent implementations, and reliance on rapidly-changing external APIs.
Firecrawl is a production-grade web scraping, crawling, and structured data extraction platform designed to feed AI agents and LLMs with clean, markdown-formatted web data. The system exposes a multi-version REST API (v0, v1, v2) with endpoints for scraping single URLs, crawling entire sites, mapping site structures, searching the web, extracting structured JSON, monitoring page changes, and running autonomous AI agents. It supports multiple output formats including markdown, HTML, screenshots, PDF parsing, product/menu extraction, and branding detection. The architecture is a microservices-style monolith with a Node.js/Express API server at its core, backed by Redis for job queues and caching, PostgreSQL for persistent state, ClickHouse for analytics, Bigtable for job access logs, and GCS for blob storage. Background workers handle crawl orchestration, PDF processing, webhook delivery, and billing via RabbitMQ and FoundationDB queues. The platform integrates with external services for threat protection (Google Web Risk, Zscaler), billing (Autumn), search (SearXNG, DuckDuckGo, X Search), and LLM-based extraction (OpenAI, Anthropic). Nine official SDKs (Python, JavaScript/TypeScript, Go, Java, .NET, Ruby, PHP, Rust, Elixir) wrap the API with typed models, error handling, and async support, making it accessible across ecosystems. The codebase is extensive and well-tested, with over 2,000 files spanning controllers, scrapers, transformers, services, and utilities, but shows signs of technical drift through deprecated v0/v1 endpoints, multiple parallel implementations of similar logic (e.g., extract pipelines), and configuration sprawl across environment variables.
ScrapeGraphAI is a Python library that extracts structured data from websites and local documents using LLM-driven graph pipelines. Instead of a traditional web scraper with hardcoded selectors, users define extraction through natural-language prompts and optional Pydantic schemas; the library then executes a directed graph of nodes such as fetching, parsing, chunking, and answer generation. The public API is a set of graph classes (SmartScraperGraph, SearchGraph, JSONScraperGraph, CSVScraperGraph, SpeechGraph, ScriptCreatorGraph, etc.) in scrapegraphai/graphs, each composing reusable nodes from scrapegraphai/nodes and running them on a BaseGraph execution engine. It is consumed programmatically in Python and supports many LLM providers (OpenAI, Ollama, Groq, Mistral, Anthropic, DeepSeek, xAI, NVIDIA, MiniMax) and multiple fetching backends (Playwright, Selenium, Chromium, BrowserBase, Scrape.do). The architecture is a lightweight DAG framework: nodes have typed inputs/outputs, can branch conditionally, iterate, merge results, and accumulate cost/token metadata; BaseGraph walks the graph, handles state, and collects execution information. The repository also contains document loaders, model token/cost registries, prompt templates, telemetry, proxy rotation, screenshot-scraping utilities, a GraphBuilder that constructs graphs from natural language, and integrations such as Burr and Indexify. While the README aggressively promotes a commercial hosted API at scrapegraphai.com, the repository itself is a self-contained open-source library with examples and tests. The codebase is generally well structured and modular, with clear separation between graphs, nodes, prompts, helpers, utils, and models. It shows signs of rapid growth: some typos, duplicated test fixtures, hardcoded model pricing tables, a Python version mismatch between pyproject and Dockerfile, and dependency fragility around LangChain and optional browser/OCR packages. These issues are manageable but give the codebase a medium technical drift. Overall it is an active, useful library for LLM-based extraction, and several utilities are good candidates for reuse outside the project.
CloakBrowser is a commercial stealth browser automation library that wraps Playwright (Python, JavaScript, .NET) with a custom patched Chromium binary. Its core purpose is to evade bot detection by spoofing browser fingerprints, providing human-like interaction (Bezier mouse moves, realistic typing with typos, smooth scrolling), and managing licensed binary downloads with cryptographic verification. The architecture is a multi-language SDK sharing identical logic across Python, TypeScript, and C#, with a common human interaction engine, proxy/GeoIP integration, and license enforcement via denial files and server validation. Key entry points are the `launch()` and `humanize()` APIs, CLI diagnostics (`info`, `login`, `install`), and the `cloakserve` CDP multiplexer. External dependencies include Playwright/Puppeteer, MaxMind GeoLite2 database, Ed25519 signature verification (BouncyCastle), and a license validation server. The codebase demonstrates high engineering discipline with extensive cross-language parity tests and security-conscious binary distribution, but carries drift risks from triple-language duplication and reliance on Playwright internal APIs.
imghash is a Node.js library for computing perceptual image hashes (pHash) using the block mean value based hashing method. It supports PNG, JPEG, BMP, and WebP image formats, and can accept file paths or Buffers as input. The library provides a simple async API to generate hashes in hex or binary format, with configurable bit lengths. It includes utility functions for converting between hex and binary representations. The codebase is written in TypeScript, well-tested with unit and e2e tests, and follows modern JavaScript practices. It handles edge cases such as palette-based PNGs and problematic JPEG images that cause longjmp errors in WebAssembly decoders by falling back to an alternative JPEG library.
This repository provides a Python library for converting web pages into text optimized for Large Language Model (LLM) input. It fetches HTML content using Playwright (headless browser) and processes it to extract meaningful text while preserving structural information like tables, links, and images in a text format. The core processing logic converts HTML tables to markdown, handles image and link preservation (or removal), and strips unnecessary tags (script, style) to reduce noise. The library is designed as a preprocessing step before feeding web content to LLMs for tasks such as data extraction or summarization. It includes three main modules: HTML fetching, HTML processing, and a combined convenience function.