Crawlee is a comprehensive web scraping and crawling library for Node.js/TypeScript that provides a unified interface for both HTTP-based crawling (using got-scraping with Cheerio/JSDOM/Linkedom) and browser-based crawling (using Playwright/Puppeteer). It features a modular architecture with separate packages for core functionality, specific crawlers, storage abstractions, browser pooling, and utilities. The library handles request queuing, automatic concurrency scaling based on system resources, session management with cookie persistence and fingerprinting, proxy rotation, and robust error handling. It includes built-in mechanisms to avoid bot detection and provides storage abstractions for datasets (structured data), key-value stores (binary data), and request queues (URL frontier). Designed for reliability and scalability, Crawlee supports both local execution and deployment on the Apify platform, with extensive documentation, examples, and CLI tools for project scaffolding.
Scrapy is a mature, production-grade web scraping and crawling framework for Python. It provides a complete pipeline for downloading web pages, extracting structured data (via XPath/CSS selectors), and exporting it in multiple formats (JSON, CSV, XML, etc.). The architecture is event-driven, built on Twisted with full asyncio integration, and consists of an execution engine, downloader, scheduler, spider middleware, downloader middleware, item pipelines, and extensions. Key entry points include the CLI (`scrapy crawl`, `scrapy shell`, `scrapy startproject`), the `CrawlerProcess`/`CrawlerRunner` classes for programmatic use, and the `Spider` base class for user-defined spiders. Data flows from HTTP requests through the downloader (with support for HTTP/1.1, HTTP/2, FTP, and experimental httpx), through spider callbacks, into item pipelines, and finally to feed exporters or custom storage backends. External dependencies include Twisted, lxml, parsel, w3lib, protego (robots.txt), queuelib, and optional integrations like boto3 (S3), google-cloud-storage, Pillow (image processing), and mitmproxy (testing). The framework is highly configurable via a priority-based settings system and supports plugins (addons, middlewares, pipelines, extensions).
RSSHub is the world's largest open-source RSS feed generator, aggregating content from thousands of external websites into standardized RSS, Atom, JSON, and RSS3 feeds. It is a TypeScript application built on the Hono web framework, with a modular architecture where each source (news site, social platform, university notice board, government agency, etc.) is implemented as a self-contained 'route' module that fetches pages or APIs, parses HTML with Cheerio, and optionally renders rich descriptions using JSX. The project supports multiple deployment targets including Node.js, Docker, Vercel, and Cloudflare Workers, and integrates caching (Redis/LRU), middleware for access control, ETag/conditional requests, error tracking (Sentry, Honeybadger), and anti-hotlinking. The codebase is highly organized with consistent route definitions, extensive utility functions (date parsing, HTTP caching, header generation), and a strong emphasis on type safety. While the sheer volume of routes (thousands) introduces maintenance burden and some inconsistency (legacy `got` vs modern `ofetch`), the overall architecture is clean, well-tested for core infrastructure, and exhibits strong engineering practice. This is not a data-research repository; it is a production-grade web service for feed aggregation.
Scrapling is a high-performance Python web scraping library that provides a unified interface for static HTTP requests (via curl_cffi with TLS fingerprint impersonation), dynamic browser automation (Playwright), and stealthy browser sessions with Cloudflare Turnstile/interstitial bypass. It features an adaptive HTML parser (Selector) that can automatically relocate elements after website structure changes using similarity scoring, stored in a SQLite-backed storage system. The library includes a full spider/crawler framework (Spider, CrawlSpider, SitemapSpider, etc.) with concurrent requests, pause/resume via checkpoints, AutoThrottle, proxy rotation, and streaming output. Additionally, it offers a CLI tool for quick extraction, an interactive IPython shell, a Scrapy integration decorator, and an MCP server that exposes 13 scraping tools for AI agents (Claude, Cursor). The codebase is well-structured with clear separation of concerns, extensive type hints, and 92% test coverage. While the library is actively maintained and uses modern Python features, it has some code duplication between sync/async implementations and relies on heavy external dependencies (Playwright, curl_cffi, lxml) that require careful version management.
Maigret is a username-based OSINT reconnaissance tool that checks across thousands of websites for accounts associated with a given username, extracts profile information via the socid_extractor library, and generates multi-format reports (HTML, PDF, CSV, JSON, Markdown, XMind, Neo4j, etc.). It operates as a CLI tool, an embeddable Python library, a Flask web application with live scan streaming, and a Docker image. The architecture is a monolith with a clean separation between site database management (MaigretDatabase/MaigretSite models), asynchronous checking engine (AsyncioQueueGeneratorExecutor, checkers for aiohttp, curl_cffi, Cloudflare bypass), extraction enrichment, and report generation. External dependencies include aiohttp, socid_extractor, curl_cffi, aiodns, and potentially FlareSolverr and OpenAI-compatible APIs (optional). The tool supports proxy/Tor/I2P routing, self-validation of site configurations, and auto-update of its site database. It is actively maintained, with frequent releases and a large community, and is used by professionals in OSINT, academic research, cryptocurrency investigations, and privacy analysis.
Crawlee is a production-grade Python web scraping and browser automation framework developed by Apify. It provides a unified async API for both static HTTP crawling (BeautifulSoup, Parsel, raw HTTP) and headless browser automation (Playwright, Stagehand), with an adaptive mode that dynamically chooses between them based on page characteristics. The architecture is modular: a context pipeline chains pre-navigation hooks, HTTP execution (via pluggable clients like httpx, curl-impersonate, or impit), parsing, and post-processing before dispatching to user-defined router handlers. Storage is abstracted behind Dataset, KeyValueStore, and RequestQueue interfaces with multiple backends (memory, filesystem, SQL, Redis). The framework includes autoscaling, session management, proxy rotation, browser fingerprint generation for anti-blocking, OpenTelemetry instrumentation, and a CLI for project scaffolding. It is designed for large-scale crawling with state persistence across runs and supports deployment on Apify, AWS Lambda, and Google Cloud Run.
Mechanize is a mature Ruby library for automating web interactions. It acts as a programmable web browser, enabling users to fetch pages, follow links, submit forms, manage cookies, handle authentication (Basic, Digest, NTLM), and support content compression (gzip, deflate, brotli, zstd). The library wraps `net-http-persistent` for connection management and `nokogiri` for HTML/XML parsing. Key architectural patterns include a pluggable parser system that maps MIME types to response handlers (e.g., `Mechanize::Page` for HTML, `Mechanize::Image`, `Mechanize::Download`), an element matching mixin for filtering links and forms, a robust cookie jar, and an authentication store. Its test suite (over 150 test files) uses a WEBrick-based mock server and servlets to simulate realistic scenarios without network calls. The library is widely used for scraping, testing, and automation in Ruby ecosystems.
Notte is a full-stack web AI agent framework and browser automation SDK. The implementation is a multi-package Python monorepo: notte-core provides shared domain models, actions, error hierarchies, telemetry, and configuration; notte-browser implements Playwright/Patchright browser control, DOM snapshotting, scraping pipelines, and form-filling heuristics; notte-agent contains the LLM-driven agent loop, deterministic fallback, and workflow execution; notte-llm wraps LiteLLM with structured completions, retries, and a file-based prompt library; notte-sdk is the client for the hosted Notte cloud API with remote sessions, agents, vaults, personas, and serverless functions; notte-integrations adds notifiers and third-party CDP session providers. Primary entry points are `NotteClient` for cloud usage, `NotteSession` for local browser automation, and `Agent`/`AgentFallback` for autonomous and hybrid scripted-agent workflows.
Colly is a mature, high-performance web scraping framework for Go, providing a clean, callback-based API for crawling and extracting structured data from HTML and XML. It handles HTTP requests, session management, caching, robots.txt compliance, proxy rotation, rate limiting, and parallel scraping out of the box. The library is built around a Collector object that orchestrates configurable callbacks (OnRequest, OnResponse, OnHTML, OnXML, OnError, OnScraped) and supports chaining, context propagation, and request queueing for large-scale scraping jobs. The codebase is well-structured, thoroughly tested, and follows Go idioms, making it a reliable choice for production scraping tasks. It is not a data research project; it is a general-purpose scraping library.
search-engine-parser is a Python library and CLI tool designed to scrape search engines for structured results. It provides a base class for search engine implementations with specific parsers for engines like Google, Bing, Yahoo, DuckDuckGo, and others. The tool supports both synchronous and asynchronous HTTP requests, includes caching to avoid repeated requests, and uses user-agent rotation to prevent blocking. Key features include a command-line interface (pysearch) for terminal use and a programmatic API for integration into other projects, returning results as SearchItem objects with convenient access to titles, links, and descriptions.
Firecrawl is a production-grade web scraping, crawling, and structured data extraction platform designed to feed AI agents and LLMs with clean, markdown-formatted web data. The system exposes a multi-version REST API (v0, v1, v2) with endpoints for scraping single URLs, crawling entire sites, mapping site structures, searching the web, extracting structured JSON, monitoring page changes, and running autonomous AI agents. It supports multiple output formats including markdown, HTML, screenshots, PDF parsing, product/menu extraction, and branding detection. The architecture is a microservices-style monolith with a Node.js/Express API server at its core, backed by Redis for job queues and caching, PostgreSQL for persistent state, ClickHouse for analytics, Bigtable for job access logs, and GCS for blob storage. Background workers handle crawl orchestration, PDF processing, webhook delivery, and billing via RabbitMQ and FoundationDB queues. The platform integrates with external services for threat protection (Google Web Risk, Zscaler), billing (Autumn), search (SearXNG, DuckDuckGo, X Search), and LLM-based extraction (OpenAI, Anthropic). Nine official SDKs (Python, JavaScript/TypeScript, Go, Java, .NET, Ruby, PHP, Rust, Elixir) wrap the API with typed models, error handling, and async support, making it accessible across ecosystems. The codebase is extensive and well-tested, with over 2,000 files spanning controllers, scrapers, transformers, services, and utilities, but shows signs of technical drift through deprecated v0/v1 endpoints, multiple parallel implementations of similar logic (e.g., extract pipelines), and configuration sprawl across environment variables.
Twikit is an asynchronous Python library that provides a comprehensive, unofficial client for the Twitter (X) API. It allows users to perform a wide range of Twitter operations—such as posting tweets, searching for tweets and users, managing bookmarks and lists, sending direct messages, uploading media, and streaming real-time events—without requiring an official API key. The library achieves this by reverse-engineering Twitter's internal GraphQL and REST v1.1 endpoints, handling authentication via login flows or guest tokens, and implementing sophisticated anti-bot evasion techniques like client-side transaction ID generation (X-Client-Transaction-Id) and UI metrics obfuscation. The architecture separates concerns into GraphQL (gql) and REST (v11) client modules, with data models for tweets, users, communities, and more. It includes optional CAPTCHA solving and cookie persistence for session reuse. The codebase is well-structured, actively maintained, and demonstrates deep reverse-engineering skill, particularly in the `x_client_transaction` module that mimics Twitter's client-side telemetry.
ScrapeGraphAI is a Python library that extracts structured data from websites and local documents using LLM-driven graph pipelines. Instead of a traditional web scraper with hardcoded selectors, users define extraction through natural-language prompts and optional Pydantic schemas; the library then executes a directed graph of nodes such as fetching, parsing, chunking, and answer generation. The public API is a set of graph classes (SmartScraperGraph, SearchGraph, JSONScraperGraph, CSVScraperGraph, SpeechGraph, ScriptCreatorGraph, etc.) in scrapegraphai/graphs, each composing reusable nodes from scrapegraphai/nodes and running them on a BaseGraph execution engine. It is consumed programmatically in Python and supports many LLM providers (OpenAI, Ollama, Groq, Mistral, Anthropic, DeepSeek, xAI, NVIDIA, MiniMax) and multiple fetching backends (Playwright, Selenium, Chromium, BrowserBase, Scrape.do). The architecture is a lightweight DAG framework: nodes have typed inputs/outputs, can branch conditionally, iterate, merge results, and accumulate cost/token metadata; BaseGraph walks the graph, handles state, and collects execution information. The repository also contains document loaders, model token/cost registries, prompt templates, telemetry, proxy rotation, screenshot-scraping utilities, a GraphBuilder that constructs graphs from natural language, and integrations such as Burr and Indexify. While the README aggressively promotes a commercial hosted API at scrapegraphai.com, the repository itself is a self-contained open-source library with examples and tests. The codebase is generally well structured and modular, with clear separation between graphs, nodes, prompts, helpers, utils, and models. It shows signs of rapid growth: some typos, duplicated test fixtures, hardcoded model pricing tables, a Python version mismatch between pyproject and Dockerfile, and dependency fragility around LangChain and optional browser/OCR packages. These issues are manageable but give the codebase a medium technical drift. Overall it is an active, useful library for LLM-based extraction, and several utilities are good candidates for reuse outside the project.
WaterCrawl is a full-stack web crawling and data extraction platform, built as a SaaS-style multi-tenant application. The backend is implemented in Django 5.1 with Django REST Framework, providing REST APIs for crawl requests (single/batch URL), web search (Google Custom Search integration), sitemap generation, and proxy server management. Asynchronous task execution is handled by Celery workers that invoke Scrapy spiders (SiteScrapper, GoogleCustomSearchScrapper, SitemapScrapper) and optionally use Playwright for JavaScript rendering. Real-time status updates are pushed to the frontend via Redis Pub/Sub and Server-Sent Events (SSE). The frontend is a React SPA (with TypeScript, Tailwind CSS, react-hook-form) that provides dashboard pages for configuring and monitoring crawl/search/sitemap jobs, managing teams and API keys, viewing usage statistics, and handling subscription billing through Stripe. Authentication supports JWT (simplejwt) and OAuth (Google, GitHub), with email verification and team invitation flows. The application stores extracted content (markdown, HTML, attachments) in MinIO or S3-compatible storage, and metadata in PostgreSQL.
xword-dl is a command-line tool that downloads crossword puzzles from over 30 online outlets and saves them in the standard .puz format. The codebase implements a plugin-based architecture where each outlet (e.g., New York Times, Guardian, Los Angeles Times) has its own downloader subclassing a base class. The main entry point (xword_dl.py) discovers plugins via package walking, parses command-line arguments, and dispatches to the appropriate downloader. The download flow consists of locating the puzzle URL (by latest or by date), fetching solver data, parsing the outlet-specific format (JSON, XML, or obfuscated payload) into a unified Puzzle object, sanitizing content for the .puz format, and saving the file. Notable technical challenges handled include deobfuscating AmuseLabs' proprietary encoding using a brute-force BFS key search, authenticating with NYT's subscription API, and translating diverse grid/clue representations into a standardized binary format. The project is actively maintained with pinned dependencies, GitHub Actions CI, and a well-documented plugin pattern. While it lacks unit tests and some downloaders are disabled due to anti-scraping measures, the core architecture is clean and the deobfuscation module is a standout reusable asset.
CloudStream is an Android media aggregation application that streams and downloads movies, TV series, and anime via a plugin-based provider system. The app has no built-in content; all media sources are provided by community extensions implementing `MainAPI` and `ExtractorApi` interfaces. The architecture comprises a Kotlin Multiplatform library defining plugin APIs and an Android app with ExoPlayer/Media3 playback, download manager, sync with tracking services (MAL, AniList, Kitsu, Simkl, Trakt), multi-source subtitle fetching, Chromecast support, and adaptive phone/TV UI. Content acquisition relies on web scraping: over 100 built-in extractors parse video hosting sites, often using JavaScript deobfuscation (via a custom JS interpreter) and AES decryption. The plugin system supports repository-based distribution, versioning, and binary compatibility checks. While the codebase is actively maintained and demonstrates good engineering practices (MVVM, coroutines, custom adapters), many extractors are brittle due to external site dependencies, and some modules exhibit tight coupling and deprecated API usage.
This repository provides a Node.js HTTP service that generates TikTok API signatures (X-Bogus and X-Gnarly) required to access TikTok's data endpoints. It uses Puppeteer to run a headless Chromium browser, injects a bundled copy of TikTok's web SDK, and computes signatures by intercepting fetch requests. The service also includes a pure Node.js implementation for X-Gnarly using ChaCha20. It exposes POST /signature and POST /fetch endpoints, manages a request queue, and automatically refreshes the browser session after 500 signatures or 30 minutes. The repository includes example scripts for fetching user videos, comments, trending content, and search results, demonstrating typical TikTok data scraping workflows. While the core functionality is well-implemented, the reliance on static SDK copies and a heavy browser dependency introduces maintenance risks as TikTok updates its anti-scraping measures.
This repository provides a Python library for converting web pages into text optimized for Large Language Model (LLM) input. It fetches HTML content using Playwright (headless browser) and processes it to extract meaningful text while preserving structural information like tables, links, and images in a text format. The core processing logic converts HTML tables to markdown, handles image and link preservation (or removal), and strips unnecessary tags (script, style) to reduce noise. The library is designed as a preprocessing step before feeding web content to LLMs for tasks such as data extraction or summarization. It includes three main modules: HTML fetching, HTML processing, and a combined convenience function.
This repository is the deployment system for Internet-in-a-Box (IIAB): an offline-first learning server for schools and communities without reliable Internet. It is an Ansible configuration-management application that installs and configures a complete local network stack — nginx reverse proxy, PHP-FPM, MySQL/PostgreSQL, hostapd WiFi access point, dnsmasq DNS/DHCP, Squid proxy, captive portal, Samba, CUPS — alongside a catalog of educational services such as Kiwix (offline Wikipedia/ZIM), Kolibri, WordPress, MediaWiki, Nextcloud, Gitea, JupyterHub, Moodle, Calibre-Web, and an offline OpenStreetMap/vector-tile mapping system with geocoding. Python, PHP, and Jinja2 templates generate systemd units, nginx virtual hosts, and app configs for every deployed service. Architecture is a staged Ansible pipeline: eleven role groups (0-init through 9-local-addons, then network) are driven from top-level playbooks (iiab-stages.yml, iiab-from-cmdline.yml) invoked by shell entry points (iiab-install, iiab-configure, iiab-network). Every role follows consistent conventions: defaults in defaults/main.yml, an install.yml and enable-or-disable.yml, boolean *_install/*_enabled flags validated up front with Ansible assert tasks, install state recorded in /etc/iiab/iiab_state.yml, and disk usage audited into iiab.ini. A custom Ansible module (get_curl) performs resumable/atomic downloads used across roles. Content provisioning is data-driven: maps and ZIM catalogs are generated (roles/maps/generate-catalog.py) and fetched at install time; offline search and geometry utilities (FQR, static_search.js) are bundled into the served front end. The project is actively developed (Ubuntu 24.04 and Raspberry Pi OS CI matrices, expected-state integration tests) but carries heavy legacy baggage: a large 0-DEPRECATED-ROLES tree, numerous .unused files, Python 2 support scripts, hardcoded versions/vendor URLs, plaintext default secrets, and fragile per-distro workarounds. The reusable gems — the custom download module, the tested geographic FQR geometry utility, the client-side static geocoder, the Ansible facts script, and the pdsm proot service manager — are isolated enough to extract into new projects. Overall it is a solid, professional infrastructure-as-code project with a few genuinely reusable components, tempered by significant accumulated technical debt.
TripStar is a full-stack, multi-agent AI travel planning platform. It uses a FastAPI backend with a modular agent system (trip planner, weather, hotel agents) orchestrated by a large language model, and a Vue 3 + TypeScript frontend with a dark glassmorphism UI. The backend provides REST APIs and WebSocket real-time status updates for asynchronous trip generation tasks. It integrates data from multiple sources: Xiaohongshu (via reverse-engineered signed API and SSR scraping), Google Maps, and AMap (with automatic fallback). The LLM extracts structured attractions from scraped text, plans itineraries, and repairs malformed JSON output. The system also includes user preference memory with weighted decay and TOP-K recall, a knowledge graph module, and export to image/PDF. The frontend features interactive maps, a budget panel, multi-city support, and a floating AI chat widget. The application is Dockerized with a multi-stage build and supports Chinese, English, and Japanese through i18n. The architecture is cleanly separated into backend services (agents, API routes, memory, map dispatcher, scraper), and frontend views (Home, Landing, Result, AIChat). The map dispatcher pattern provides a unified interface for geocoding and POI search across Google and AMap, and the memory manager implements a reusable preference-learning system. However, the codebase has technical debt: heavy reliance on Xiaohongshu's undocumented API (fragile), a monolithic 42k-token CSS file from Paper Kit 2 (2017) with deprecated techniques, and a complex ~2000-line Result.vue component. Dependencies are not fully locked, and the scraper's JavaScript signature engine may break with anti-bot updates. Overall, TripStar is a solid, functional application with several reusable patterns, but it carries moderate maintenance risks due to its external dependencies and some aging code. Note: This is a hybrid application with a strong data-crawling component (Xiaohongshu scraping and LLM extraction), but the dominant character is a production-ready web application with a clear user interface and API.