$ show LearningCircuit/local-deep-research
Local Deep Research is a self-hosted, privacy-focused AI research assistant that performs deep, agentic research using multiple LLMs and web searches. It provides a full-stack web application (FastAPI + JavaScript), a programmatic API, and a Model Context Protocol (MCP) server. The system ingests user queries, decomposes them, executes parallel searches across web and academic sources (SearXNG, Google, arXiv, PubMed, Semantic Scholar, etc.), fetches and extracts content, evaluates evidence, and synthesizes reports with citations. It features per-user encrypted SQLCipher databases, a settings system with environment variable overrides, a RAG library with FAISS vector store, journal quality scoring, news subscriptions, and extensive security controls (egress policy, SSRF protection, credential scrubbing). The codebase is heavily tested with unit, integration, UI, accessibility, and security tests. It is a hybrid of a web application and a research data pipeline, but the dominant character is a production-grade application.
The LLM identified 3 files worth highlighting for reuse:
A centralized egress policy module that evaluates URLs and classifies search engines/research mode into five scopes (Adaptive, Public only, Private only, Strict, Unprotected). It includes an audit hook that enforces policy at the socket level, a classification engine, warning system, and validators. This is an isolated, reusable security pattern that could be extracted into any application needing fine-grained control over outbound data access.
A thread-safe SettingsManager that provides unified access to application settings from environment variables (LDR_*), JSON defaults, and database-backed values. It supports typed value parsing, thread-safe snapshots, env-locked protection, and golden master validation. The component is well-abstracted with a base class and could be reused in any Python application requiring robust configuration management.
A semantic chunker that applies character-based pre- and post-bounding around semantic splits to ensure chunk size constraints are met. It handles edge cases like single-chunk documents and overlapping boundaries. This is a non-obvious data preprocessing component that improves RAG quality and could be extracted for use in any text splitting pipeline.