Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
Cheerio is a fast, flexible, and elegant library for parsing and manipulating HTML and XML on the server side (Node.js). It provides a jQuery-like API for traversing and manipulating the DOM of parsed documents, designed for web scraping, data extraction, and HTML transformation tasks. The library abstracts away the underlying parser (htmlparser2 for speed, parse5 for standards compliance) via a factory pattern, offering a consistent Cheerio API regardless of parser choice. Key entry points include the main `load` function that converts HTML strings or buffers into a Cheerio instance, static methods like `html`, `xml`, `text`, and `parseHTML`, and utility functions for CSS handling, form serialization, and attribute manipulation. External dependencies include cheerio-select for CSS selector parsing, domutils for core DOM traversal, and optional HTTP fetching via undici for live URL parsing.
Crawl4AI is a production-grade, async-first web crawling and scraping library designed to transform any website into clean, LLM-ready Markdown or structured data. It serves as both a Python library and a self-hosted Docker API server, making it suitable for RAG pipelines, AI agents, and data extraction workflows. The architecture is built on pluggable strategies: extraction (CSS, XPath, LLM, Regex), content filtering (BM25, pruning, LLM), chunking (topic, regex, sentence, sliding window), deep crawling (BFS, DFS, Best-First), and proxy rotation. Key entry points include the `AsyncWebCrawler` class with `arun()`/`arun_many()` methods, configuration objects (`BrowserConfig`, `CrawlerRunConfig`), and a CLI (`crwl`). The data flow progresses from URL → browser/HTTP fetch → raw HTML → cleaned HTML → Markdown → filtered content → structured extraction, with caching and cache validation as first-class features. External dependencies include Playwright for browser automation, httpx/aiohttp for HTTP, lxml for fast parsing, and optional LLM providers (OpenAI, Anthropic, Groq, etc.) and ML models (sentence-transformers). The Docker deployment adds a FastAPI server with JWT auth, Redis-backed job queue, webhook support, MCP bridge, and a monitoring dashboard, all hardened against SSRF, XSS, and file-write attacks. While primarily an application, the repository also contains data-research elements such as adaptive crawling with embedding-based relevance scoring, anti-bot detection heuristics, and content filtering research, reflecting a hybrid character dominated by the application/library use case.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
Huginn is a self-hosted, hackable automation platform built as a Ruby on Rails application. It allows users to create and chain 'agents' that monitor, scrape, transform, and act on data from hundreds of sources (web, RSS, email, Twitter, Telegram, Slack, OpenAI, etc.) via an event-driven architecture. Agents are polymorphic Ruby classes with a consistent lifecycle: scheduled checks, event reception, and event emission. The system supports Liquid templating for dynamic configuration, OAuth services, file pointers, and background job processing via DelayedJob. The codebase is extensive with 80+ agent types, reusable concerns (WebRequestConcern, LiquidInterpolatable, FormConfigurable), and a comprehensive test suite. While primarily an application, it includes a small research dataset (ANEW sentiment norms) used by a sentiment analysis agent, but the dominant character is clearly a production automation service.
Colly is a mature, high-performance web scraping framework for Go, providing a clean, callback-based API for crawling and extracting structured data from HTML and XML. It handles HTTP requests, session management, caching, robots.txt compliance, proxy rotation, rate limiting, and parallel scraping out of the box. The library is built around a Collector object that orchestrates configurable callbacks (OnRequest, OnResponse, OnHTML, OnXML, OnError, OnScraped) and supports chaining, context propagation, and request queueing for large-scale scraping jobs. The codebase is well-structured, thoroughly tested, and follows Go idioms, making it a reliable choice for production scraping tasks. It is not a data research project; it is a general-purpose scraping library.
Firecrawl is a production-grade web scraping, crawling, and structured data extraction platform designed to feed AI agents and LLMs with clean, markdown-formatted web data. The system exposes a multi-version REST API (v0, v1, v2) with endpoints for scraping single URLs, crawling entire sites, mapping site structures, searching the web, extracting structured JSON, monitoring page changes, and running autonomous AI agents. It supports multiple output formats including markdown, HTML, screenshots, PDF parsing, product/menu extraction, and branding detection. The architecture is a microservices-style monolith with a Node.js/Express API server at its core, backed by Redis for job queues and caching, PostgreSQL for persistent state, ClickHouse for analytics, Bigtable for job access logs, and GCS for blob storage. Background workers handle crawl orchestration, PDF processing, webhook delivery, and billing via RabbitMQ and FoundationDB queues. The platform integrates with external services for threat protection (Google Web Risk, Zscaler), billing (Autumn), search (SearXNG, DuckDuckGo, X Search), and LLM-based extraction (OpenAI, Anthropic). Nine official SDKs (Python, JavaScript/TypeScript, Go, Java, .NET, Ruby, PHP, Rust, Elixir) wrap the API with typed models, error handling, and async support, making it accessible across ecosystems. The codebase is extensive and well-tested, with over 2,000 files spanning controllers, scrapers, transformers, services, and utilities, but shows signs of technical drift through deprecated v0/v1 endpoints, multiple parallel implementations of similar logic (e.g., extract pipelines), and configuration sprawl across environment variables.
Twikit is an asynchronous Python library that provides a comprehensive, unofficial client for the Twitter (X) API. It allows users to perform a wide range of Twitter operations—such as posting tweets, searching for tweets and users, managing bookmarks and lists, sending direct messages, uploading media, and streaming real-time events—without requiring an official API key. The library achieves this by reverse-engineering Twitter's internal GraphQL and REST v1.1 endpoints, handling authentication via login flows or guest tokens, and implementing sophisticated anti-bot evasion techniques like client-side transaction ID generation (X-Client-Transaction-Id) and UI metrics obfuscation. The architecture separates concerns into GraphQL (gql) and REST (v11) client modules, with data models for tweets, users, communities, and more. It includes optional CAPTCHA solving and cookie persistence for session reuse. The codebase is well-structured, actively maintained, and demonstrates deep reverse-engineering skill, particularly in the `x_client_transaction` module that mimics Twitter's client-side telemetry.
Lux is a fast and simple video downloader CLI tool written in Go. It supports downloading videos, images, and audio from over 40 websites including YouTube, Bilibili, TikTok, Instagram, Twitter, and many Chinese streaming platforms. The tool operates as a single binary with a rich set of command-line options for selecting quality, downloading playlists, resuming interrupted downloads, multi-threaded downloading, and integration with aria2 for RPC-based downloads. Architecturally, Lux follows a plugin-based extractor pattern. Each supported website has its own extractor module under `extractors/`, registered via `init()` functions into a global map keyed by domain. The main entry point is `main.go`, which parses CLI flags via `urfave/cli` and dispatches to the appropriate extractor. Extracted video metadata (streams, quality, size) is passed to the downloader, which handles HTTP requests, progress bars, file merging via ffmpeg, and optional subtitle embedding. Key data flows: URL input → domain detection → extractor selection → HTML/API scraping → stream metadata extraction → downloader (single or multi-threaded) → file output. External dependencies include ffmpeg for merging and subtitle embedding, and various third-party Go libraries for HTTP requests, HTML parsing, JavaScript execution (for TikTok signing), and progress bars. The codebase is well-organized with clear separation of concerns, extensive test coverage for individual extractors, and CI workflows that run weekly to verify extractor functionality. While many extractors rely on fragile regex-based scraping of HTML pages, the overall architecture is solid and the downloader implements robust features like resume support and configurable retry logic.
WaterCrawl is a full-stack web crawling and data extraction platform, built as a SaaS-style multi-tenant application. The backend is implemented in Django 5.1 with Django REST Framework, providing REST APIs for crawl requests (single/batch URL), web search (Google Custom Search integration), sitemap generation, and proxy server management. Asynchronous task execution is handled by Celery workers that invoke Scrapy spiders (SiteScrapper, GoogleCustomSearchScrapper, SitemapScrapper) and optionally use Playwright for JavaScript rendering. Real-time status updates are pushed to the frontend via Redis Pub/Sub and Server-Sent Events (SSE). The frontend is a React SPA (with TypeScript, Tailwind CSS, react-hook-form) that provides dashboard pages for configuring and monitoring crawl/search/sitemap jobs, managing teams and API keys, viewing usage statistics, and handling subscription billing through Stripe. Authentication supports JWT (simplejwt) and OAuth (Google, GitHub), with email verification and team invitation flows. The application stores extracted content (markdown, HTML, attachments) in MinIO or S3-compatible storage, and metadata in PostgreSQL.
NewPipe Extractor is a Java library that provides a unified abstraction for extracting structured metadata (streams, channels, playlists, comments, search results) from multiple streaming platforms including YouTube, SoundCloud, Bandcamp, PeerTube, and media.ccc.de. It reverse-engineers private APIs and scrapes HTML/JSON to produce normalized data models. The library is designed as a headless component for the NewPipe Android app but can be used independently. Its architecture features a service registry, abstract extractor classes, link handler factories for URL normalization, and a collector pattern for batch extraction with error accumulation. The codebase is well-tested with JUnit 5 and a mock/recording downloader system, but YouTube-specific extraction is fragile due to reliance on hardcoded client versions, JavaScript deobfuscation, and frequently changing InnerTube API schemas.
DevDocs is a web application that aggregates and serves API documentation for hundreds of programming languages, frameworks, and tools in a unified, searchable, offline-capable interface. The backend is a Ruby/Sinatra server that provides the static assets and a minimal API, while the frontend is a custom JavaScript single-page application with client-side search, keyboard navigation, and a service worker for offline access. The core data generation pipeline consists of Ruby scrapers that fetch documentation pages from official sources, apply a chain of filters (HTML cleaning, entry extraction, URL normalization, image optimization), and produce normalized JSON indexes and cached pages stored on disk. The architecture is largely monolithic with a strong separation between the scraping pipeline (lib/docs) and the frontend (assets/javascripts). While primarily an application, it contains a substantial data engineering component in its scraper/filter system, which is reusable and generalizable for other documentation aggregation projects.
This repository provides a Python CLI tool and Makefile pipeline to scrape all essays from Paul Graham's website (paulgraham.com), convert them to Markdown, and produce an EPUB ebook. The core logic resides in a single script (`graham.py`) that uses `requests`, `BeautifulSoup`, `html2text`, and `htmldate` to download, parse, and date-stamp each essay. A `Makefile` orchestrates the full build: creating a virtual environment, fetching essays, merging them with `pandoc`, and generating the final EPUB with a custom CSS and an iBooks dark-mode patch. The CI workflow (GitHub Actions) runs daily, detects new essays by comparing article counts, and automatically creates a release with the updated EPUB and metadata CSV. The architecture is straightforward—no web server, database, or tests—but the pipeline is robust and well-documented for its single purpose. While the scraper is tightly coupled to Paul Graham's site structure, several components (the iBooks patcher, the footnote conversion function, and the EPUB stylesheet) are cleanly isolated and reusable in other EPUB generation projects.
CF-Clearance-Scraper is a Python CLI tool designed to solve Cloudflare challenges (JavaScript, managed, and interactive) and extract the cf_clearance cookie for bypassing anti-bot protections. The application uses Zendriver for headless browser automation, dynamically generates realistic Chrome user agents, handles proxy authentication via CDP events, and provides output formats including raw cookies, cURL/Wget/aria2 commands, or JSON storage. The architecture is monolithic with a single entry point (main.py) featuring clean separation of concerns: argument parsing, browser orchestration, challenge detection/solving, and output formatting. Key strengths include robust user agent fallback mechanics and proxy handling, though the Cloudflare-solving logic is tightly coupled to Zendriver and challenge-specific heuristics.
This repository provides a Python library for converting web pages into text optimized for Large Language Model (LLM) input. It fetches HTML content using Playwright (headless browser) and processes it to extract meaningful text while preserving structural information like tables, links, and images in a text format. The core processing logic converts HTML tables to markdown, handles image and link preservation (or removal), and strips unnecessary tags (script, style) to reduce noise. The library is designed as a preprocessing step before feeding web content to LLMs for tasks such as data extraction or summarization. It includes three main modules: HTML fetching, HTML processing, and a combined convenience function.
EasySpider is a visual, no-code web scraping tool that combines an Electron desktop application, a Chrome browser extension, and a Python execution engine. Users design scraping workflows via a flowchart-based UI (Vue.js) where they define steps like opening pages, clicking elements, extracting data, looping, and conditional branching. The task definitions are saved as JSON graphs. The Python backend (ExecuteStage) loads these graphs and drives Chrome using Selenium with an undetected-chromedriver fork to evade bot detection. Extracted data can be written to CSV, Excel, JSON, or MySQL. The tool also includes OCR support (ddddocr), custom JavaScript/Python execution, and Cloudflare bypass capabilities. The architecture is a hybrid desktop app with a local HTTP server (Express) that spawns executor processes and communicates via WebSocket. The Chrome extension assists in element selection and XPath generation. While the tool is functional and popular (44k+ stars), the codebase exhibits significant duplication (e.g., identical task files across platform directories, duplicated Bootstrap and content scripts), hardcoded paths, monolithic executor scripts, and mixed-language strings. Several experimental modules (LLM, federated learning) are present but not integrated. The project is actively maintained with recent commits and packaging scripts, but technical debt and drift (Bootstrap 4, Vue 2, pinned Python dependencies) are moderate.
Pipe Viewer is a Perl-based YouTube client providing both a CLI (`pipe-viewer`) and a GTK3 GUI (`gtk-pipe-viewer`). It allows users to search, stream, download, and manage YouTube content including videos, playlists, channels, comments, and captions. The core library `WWW::PipeViewer` abstracts YouTube data extraction via direct web scraping (ytInitialData, protobuf) and the Invidious API, with yt-dlp as a fallback for streaming URLs and comments. The architecture separates network resilience, caching, and format handling into reusable Perl modules (`DiskCache`, `Proto`, `ParseJSON`, etc.), though the CLI and GUI scripts are monolithic and tightly coupled to the library. External dependencies include `mpv`/`vlc` for playback, `yt-dlp` for advanced extraction, and `ffmpeg` for conversion. The project is actively maintained but operates on fragile, unofficial YouTube interfaces, making it susceptible to upstream changes.
Elib2Ebook is a .NET 10 application that downloads books from 60+ online libraries (Litres, MyBook, BookRiver, etc.) and converts them into standard ebook formats (EPUB, FB2, CBZ, JSON, TXT). It is a dual-deployment solution: a CLI tool and a Blazor web UI, both sharing a core domain and service layer. The architecture is a layered .NET solution with Domain, DomainServices, and per-site ExternalServices projects, using a reflection-based GetterFactory to auto-discover site adapters. The system handles site-specific anti-bot measures, authentication, and pagination, then normalizes content into a common domain model before passing it to format-specific builders.
GreenResourcesManager is an Electron + Vue 3 desktop application for local-first management of personal digital media collections—games, images, videos, novels, audio, websites, and software. It provides a config-driven UI with customizable pages, tag-based filtering, batch import, and built-in readers (EPUB, PDF, comic viewer, video player). Notable features include a desktop pet gamification system (affection, hunger, earnings), a disguise mode for privacy, a plugin system, and a local HTTP API for external access. Data is stored in SQLite with legacy JSON file backup. The application is actively developed (last commit April 2026) with a sizable feature set and internationalization (zh-CN, zh-TW, en, ja). However, the codebase shows significant technical drift: near-duplicate main-process code in `electron/` and `public/js/`, incomplete migration from JSON to SQLite, hardcoded Windows-specific paths, and extensive debug logging scattered throughout. The pet subsystem is tightly coupled and the plugin system remains marked as under development. While the config-driven UI pattern and composable architecture offer reuse potential, overall craftsmanship is average—functional but with notable boilerplate and coupling.
This repository is a comprehensive collection of configuration files, JavaScript and Python scrapers, and JSON data for TVBox-like media center applications (e.g., 影视仓, OK影视). It aggregates video content from hundreds of Chinese streaming sites, cloud drives (Aliyun, Quark, 115, etc.), live TV sources, and anime platforms. The core is a set of scraping rules that follow the TVBox ecosystem's conventions, using a templating system (muban) and a shared runtime (drpy2.js) for executing JavaScript-based rules. The repository also includes standalone Python spiders (dr_py compatible) and static JSON configurations for site definitions, search endpoints, and filter parameters. While the code is functional and actively used (2257 stars, 0 open issues), it exhibits significant technical debt: hardcoded API keys, duplicated code across many files, fragile parsing logic dependent on external site structures, and embedded secrets (tokens, cookies). The repository is not a standalone application but a resource library for media aggregation, making it best suited for integration into TVBox variants. It is not a data research project; there are no notebooks, pipelines, or analytical models.