$ show ScrapeGraphAI/Scrapegraph-ai
ScrapeGraphAI is a Python library that extracts structured data from websites and local documents using LLM-driven graph pipelines. Instead of a traditional web scraper with hardcoded selectors, users define extraction through natural-language prompts and optional Pydantic schemas; the library then executes a directed graph of nodes such as fetching, parsing, chunking, and answer generation. The public API is a set of graph classes (SmartScraperGraph, SearchGraph, JSONScraperGraph, CSVScraperGraph, SpeechGraph, ScriptCreatorGraph, etc.) in scrapegraphai/graphs, each composing reusable nodes from scrapegraphai/nodes and running them on a BaseGraph execution engine. It is consumed programmatically in Python and supports many LLM providers (OpenAI, Ollama, Groq, Mistral, Anthropic, DeepSeek, xAI, NVIDIA, MiniMax) and multiple fetching backends (Playwright, Selenium, Chromium, BrowserBase, Scrape.do). The architecture is a lightweight DAG framework: nodes have typed inputs/outputs, can branch conditionally, iterate, merge results, and accumulate cost/token metadata; BaseGraph walks the graph, handles state, and collects execution information. The repository also contains document loaders, model token/cost registries, prompt templates, telemetry, proxy rotation, screenshot-scraping utilities, a GraphBuilder that constructs graphs from natural language, and integrations such as Burr and Indexify. While the README aggressively promotes a commercial hosted API at scrapegraphai.com, the repository itself is a self-contained open-source library with examples and tests. The codebase is generally well structured and modular, with clear separation between graphs, nodes, prompts, helpers, utils, and models. It shows signs of rapid growth: some typos, duplicated test fixtures, hardcoded model pricing tables, a Python version mismatch between pyproject and Dockerfile, and dependency fragility around LangChain and optional browser/OCR packages. These issues are manageable but give the codebase a medium technical drift. Overall it is an active, useful library for LLM-based extraction, and several utilities are good candidates for reuse outside the project.
The LLM identified 5 files worth highlighting for reuse:
A safe_deepcopy implementation that handles objects deepcopy cannot normally process, such as boto3 clients, pydantic BaseModel instances, and objects without __dict__. It is a self-contained, difficult-to-get-right utility that can be dropped into any Python project that needs resilient state copying in agent or pipeline runtimes.
A tolerant JSON output parser that repairs malformed LLM responses, including doubled braces and other JSON quirks common in weaker models. This pattern is directly reusable in any LLM application that needs to convert noisy model output into structured data without failing the whole pipeline.
Typed dataclasses and helpers for interacting with the OpenAI Batch API, abstracting job submission, result collection, and status tracking. It is a self-contained integration layer that can be extracted for cost-efficient bulk LLM processing.
A multi-engine web search utility that normalizes queries across DuckDuckGo, Bing, SearXNG, and Serper, with rate limiting, proxy support, and PDF-filtering options. The engine-agnostic interface and rate-limit decorator make it reusable for any search-enabled data collection pipeline.
A layered HTML reduction utility that removes boilerplate, strips attributes, and optionally truncates text to produce LLM-friendly inputs. The level-based approach (0-2) is a reusable pattern for converting raw HTML into compact, token-efficient representations.