$ show assafelovic/gpt-researcher
GPT Researcher is an autonomous research agent that produces comprehensive, cited reports from web and local sources. In production, it operates as a full-stack web application: a FastAPI backend (main.py, backend/server/app.py) exposes REST and WebSocket endpoints for initiating research, streaming progress, and retrieving reports. A Next.js frontend (frontend/nextjs/) provides the user interface, and a lightweight vanilla JS frontend (frontend/scripts.js) is also available. The core Python package (gpt_researcher/) can be used programmatically via the GPTResearcher class or as a CLI tool (cli.py). Architecture follows a planner-executor pattern: given a research task, the system generates sub-queries, executes them in parallel across 20+ search engine wrappers (Tavily, Google, Bing, Brave, DuckDuckGo, ArXiv, PubMed, etc.), scrapes the resulting URLs using multiple backends (BeautifulSoup, browser automation, PyMuPDF, FireCrawl), filters and compresses the context using embedding or keyword-based methods, and finally generates a structured report with citations. The system supports multiple report types (summary, detailed, deep research) and multi-agent orchestrations (LangGraph, AG2) for more complex workflows. Key data flows: user query to sub-query generation (LLM) to parallel search across configured retrievers to URL scraping to context compression to source curation to report generation (LLM) to output as markdown/PDF/DOCX. Real-time progress is streamed via WebSocket. External dependencies include LLM providers (OpenAI, Anthropic, Google, Ollama, etc.), search APIs (Tavily, SerpAPI, etc.), and optional services (MCP servers, vector stores, image generation APIs). The codebase demonstrates strong engineering practices: extensive test suite (200+ tests), plugin architecture for retrievers, configurable context filters, cost tracking, and evaluation frameworks for quality and hallucination. However, it carries maintenance risks from its large dependency surface, multiple parallel agent implementations, and reliance on rapidly-changing external APIs.
The LLM identified 7 files worth highlighting for reuse:
JevContextCompressor: a reusable relevance filter using TypeSafe API for scoring passages. Configurable chunk size, min score, and cost callback. Demonstrates non-obvious skill in context filtering and can be extracted for any project needing semantic relevance filtering.
WorkerPool with GlobalRateLimiter: a generic async worker pool with global rate limiting. Reusable pattern for concurrent task execution with rate control, applicable to any async Python project.
validate_url and is_safe_url: SSRF/local-file protection utilities. Demonstrates security awareness and can be reused in any project that handles user-supplied URLs to prevent server-side request forgery.
MCPClientManager: manages MCP client lifecycle with config conversion and tool retrieval. Demonstrates a clean abstraction for integrating external tools via Model Context Protocol, reusable for any MCP-based integration.
Quality evaluation metrics (citation faithfulness, source diversity, source authority, subtopic coverage, unsupported claim rate). Well-structured metric interface with perturbation validation. Reusable for any RAG evaluation pipeline.
Retriever plugin system: entry-point-based extensibility for third-party retrievers. Demonstrates a clean plugin architecture using Python entry points, allowing easy addition of new search engines without modifying core code.
Config class: singleton configuration with environment variable override, type coercion, deprecated attribute handling. Reusable pattern for managing complex configurations with multiple sources and backward compatibility.