GPT Researcher is an autonomous research agent that produces comprehensive, cited reports from web and local sources. In production, it operates as a full-stack web application: a FastAPI backend (main.py, backend/server/app.py) exposes REST and WebSocket endpoints for initiating research, streaming progress, and retrieving reports. A Next.js frontend (frontend/nextjs/) provides the user interface, and a lightweight vanilla JS frontend (frontend/scripts.js) is also available. The core Python package (gpt_researcher/) can be used programmatically via the GPTResearcher class or as a CLI tool (cli.py). Architecture follows a planner-executor pattern: given a research task, the system generates sub-queries, executes them in parallel across 20+ search engine wrappers (Tavily, Google, Bing, Brave, DuckDuckGo, ArXiv, PubMed, etc.), scrapes the resulting URLs using multiple backends (BeautifulSoup, browser automation, PyMuPDF, FireCrawl), filters and compresses the context using embedding or keyword-based methods, and finally generates a structured report with citations. The system supports multiple report types (summary, detailed, deep research) and multi-agent orchestrations (LangGraph, AG2) for more complex workflows. Key data flows: user query to sub-query generation (LLM) to parallel search across configured retrievers to URL scraping to context compression to source curation to report generation (LLM) to output as markdown/PDF/DOCX. Real-time progress is streamed via WebSocket. External dependencies include LLM providers (OpenAI, Anthropic, Google, Ollama, etc.), search APIs (Tavily, SerpAPI, etc.), and optional services (MCP servers, vector stores, image generation APIs). The codebase demonstrates strong engineering practices: extensive test suite (200+ tests), plugin architecture for retrievers, configurable context filters, cost tracking, and evaluation frameworks for quality and hallucination. However, it carries maintenance risks from its large dependency surface, multiple parallel agent implementations, and reliance on rapidly-changing external APIs.
Autoresearch is a self-contained autonomous ML experimentation framework created by Andrej Karpathy. Its core premise is that an AI agent (e.g., Claude or Codex) modifies a single training script (`train.py`), runs a fixed 5-minute training experiment on a single GPU, evaluates validation bits-per-byte (BPB), and keeps or discards changes based on whether the metric improved — all without human intervention. The human programs the agent via `program.md`, which serves as a lightweight instruction set for the experiment loop. The codebase has two fixed files and one editable file. `prepare.py` handles data downloading (parquet shards from HuggingFace), BPE tokenizer training (via rustbpe), a custom BOS-aligned packed dataloader with best-fit document packing (zero padding), and a fixed BPB evaluation harness. `train.py` contains a GPT model implementation with RoPE, sliding window attention (SSSL pattern), ResFormer value embeddings, RMS normalization, and a hybrid Muon+AdamW optimizer with per-parameter-type learning rate scheduling. The model uses `torch.compile` and runs in bf16 with Flash Attention 3. The `analysis.ipynb` notebook provides post-hoc visualization of experiment results from `results.tsv`. The project is explicitly designed for a single NVIDIA GPU (H100 tested), requires no distributed training infrastructure, and pins all dependencies including PyTorch 2.9.1 with CUDA 12.8. The architecture is deliberately minimal — the entire research loop runs in two Python files with roughly 1,600 lines of non-comment code total.