Exa MCP Server is a Model Context Protocol server that connects AI agents to Exa's web search and research capabilities. It provides a standardized interface for agents to perform web searches, fetch webpage content, run multi-step research agents, and access other Exa functionalities. The architecture features clear separation of concerns with API handlers for Vercel deployment, core MCP server logic, tool implementations for each Exa capability, and utility functions for authentication, logging, and error handling. Key entry points include stdio CLI for local usage and Vercel HTTP functions for cloud deployment. The system handles authentication via API keys or OAuth, implements rate limiting for free tier users, and includes mechanisms to prevent API key leakage between requests.
This repository implements an MCP (Model Context Protocol) server that bridges AI assistants with SearXNG, a privacy-respecting metasearch engine. It exposes four primary tools: `searxng_web_search` for web queries, `searxng_search_suggestions` for autocomplete, `searxng_instance_info` for capability discovery, and `web_url_read` for fetching and converting web pages to markdown. The server runs in two transport modes: STDIO (default) for simple MCP integration, and HTTP for stateful/stateless remote access with optional OAuth and static bearer authentication. The architecture is modular and security-focused. Core modules include a proxy layer with per-type agent selection, SSRF protection via private IP blocking and DNS rebinding detection, a browser solver chain (FlareSolverr/Byparr) for challenge solving, worker-threaded PDF extraction, and multi-instance failover/fanout for SearXNG replicas. All credential-bearing output (logs, errors, MCP resources) is redacted by a centralized sanitizer. Extensive environment-variable configuration (~40+ knobs) controls caching, timeouts, rate limiting, TLS, and result formatting. Highlights include a strict integer parser that rejects unsafe or malformed values, an LFU cache with age tie-breakers and configurable TTL, and a comprehensive URL security module covering both IPv4 and IPv6 private ranges. The test suite achieves over 90% line coverage and includes integration, unit, fuzz, and e2e tests, with documentation tests that verify README and configuration docs match source contracts. The project is actively maintained with frequent releases, pinned dependencies, and a verification gate that enforces zero-vulnerability audits before publishing.
This is a mature, actively maintained AI agent skill called 'last30days' that enables agents (Claude Code, Codex, Grok, Gemini CLI, MCP hosts) to research any topic by aggregating content from 20+ social and news platforms (Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, arXiv, Techmeme, Digg, and more) over a rolling 30-day window. The system is primarily a CLI engine (Python) with a companion Go MCP server for Claude Desktop. It employs a sophisticated pipeline: query planning by intent, parallel retrieval from multiple backends (keyless APIs, paid scrapers, CLI tools, LLM-based search), source-specific normalization, engagement-weighted relevance scoring, cross-source clustering, and deterministic fusion (weighted RRF) with per-author caps and diversity floors. Output is rendered as structured JSON, compact text, or self-contained HTML with optional publishing and Atom feeds. The codebase emphasizes security (credential hygiene, no browser cookie reads in unattended mode), determinism (fixture-based evaluation harness with grounding/recency/coherence scores), and graceful degradation (per-source fallback chains, never-raise error envelopes). While classified as an application (it's a skill/plugin for agent hosts), it exhibits strong data engineering traits: source adapters, normalization pipelines, scoring models, and an offline evaluation framework.
Firecrawl is a production-grade web scraping, crawling, and structured data extraction platform designed to feed AI agents and LLMs with clean, markdown-formatted web data. The system exposes a multi-version REST API (v0, v1, v2) with endpoints for scraping single URLs, crawling entire sites, mapping site structures, searching the web, extracting structured JSON, monitoring page changes, and running autonomous AI agents. It supports multiple output formats including markdown, HTML, screenshots, PDF parsing, product/menu extraction, and branding detection. The architecture is a microservices-style monolith with a Node.js/Express API server at its core, backed by Redis for job queues and caching, PostgreSQL for persistent state, ClickHouse for analytics, Bigtable for job access logs, and GCS for blob storage. Background workers handle crawl orchestration, PDF processing, webhook delivery, and billing via RabbitMQ and FoundationDB queues. The platform integrates with external services for threat protection (Google Web Risk, Zscaler), billing (Autumn), search (SearXNG, DuckDuckGo, X Search), and LLM-based extraction (OpenAI, Anthropic). Nine official SDKs (Python, JavaScript/TypeScript, Go, Java, .NET, Ruby, PHP, Rust, Elixir) wrap the API with typed models, error handling, and async support, making it accessible across ecosystems. The codebase is extensive and well-tested, with over 2,000 files spanning controllers, scrapers, transformers, services, and utilities, but shows signs of technical drift through deprecated v0/v1 endpoints, multiple parallel implementations of similar logic (e.g., extract pipelines), and configuration sprawl across environment variables.
ScrapeGraphAI is a Python library that extracts structured data from websites and local documents using LLM-driven graph pipelines. Instead of a traditional web scraper with hardcoded selectors, users define extraction through natural-language prompts and optional Pydantic schemas; the library then executes a directed graph of nodes such as fetching, parsing, chunking, and answer generation. The public API is a set of graph classes (SmartScraperGraph, SearchGraph, JSONScraperGraph, CSVScraperGraph, SpeechGraph, ScriptCreatorGraph, etc.) in scrapegraphai/graphs, each composing reusable nodes from scrapegraphai/nodes and running them on a BaseGraph execution engine. It is consumed programmatically in Python and supports many LLM providers (OpenAI, Ollama, Groq, Mistral, Anthropic, DeepSeek, xAI, NVIDIA, MiniMax) and multiple fetching backends (Playwright, Selenium, Chromium, BrowserBase, Scrape.do). The architecture is a lightweight DAG framework: nodes have typed inputs/outputs, can branch conditionally, iterate, merge results, and accumulate cost/token metadata; BaseGraph walks the graph, handles state, and collects execution information. The repository also contains document loaders, model token/cost registries, prompt templates, telemetry, proxy rotation, screenshot-scraping utilities, a GraphBuilder that constructs graphs from natural language, and integrations such as Burr and Indexify. While the README aggressively promotes a commercial hosted API at scrapegraphai.com, the repository itself is a self-contained open-source library with examples and tests. The codebase is generally well structured and modular, with clear separation between graphs, nodes, prompts, helpers, utils, and models. It shows signs of rapid growth: some typos, duplicated test fixtures, hardcoded model pricing tables, a Python version mismatch between pyproject and Dockerfile, and dependency fragility around LangChain and optional browser/OCR packages. These issues are manageable but give the codebase a medium technical drift. Overall it is an active, useful library for LLM-based extraction, and several utilities are good candidates for reuse outside the project.