$ show lfnovo/open-notebook
Open Notebook is a self-hosted, AI-powered research assistant and knowledge management application. It allows users to collect sources (web pages, uploaded documents, text snippets), extract structured insights via LLM-driven transformations, organize content into notebooks, perform hybrid text/vector search, and generate AI podcasts with multi-speaker support using TTS models. The backend is built on Python with FastAPI and LangGraph for orchestration, SurrealDB for persistence (with a custom migration system), and the Esperanto library for multi-provider AI integration (OpenAI, Anthropic, Google, Ollama, and many others). The frontend is a modern Next.js application with React Query, i18n support for 14 locales, and a component library based on Radix UI/shadcn. The architecture follows a monolith-with-background-workers pattern: the FastAPI process handles both synchronous API requests and, via the surreal-commands library, asynchronous background jobs for content processing, embedding generation, and podcast generation. The codebase is well-structured with clear separation between API routers, domain models, utility modules, and frontend components. It exhibits strong security awareness—includes SSRF protection via DNS pinning, path containment for file uploads, Fernet encryption for stored credentials, and input validation throughout. The test suite is extensive (>100 test files) covering API endpoints, domain logic, error handling, and edge cases. There are some maintenance risks related to tight coupling with SurrealDB and the externally-maintained Esperanto/content-core/podcast-creator libraries, but overall the codebase is actively maintained and of professional quality.
The LLM identified 4 files worth highlighting for reuse:
Provides `validate_url` and `prepare_pinned_http_target` functions that protect against Server-Side Request Forgery (SSRF) attacks. `prepare_pinned_http_target` resolves a hostname to its IP at request time, constructs a direct IP URL, and sets the original hostname as the HTTP Host header and TLS SNI extension, closing the DNS rebinding TOCTOU window. This pattern is reusable in any project that needs to make safe outbound HTTP requests to user-supplied URLs. It also blocks link-local, multicast, and AWS IMDSv6 addresses while allowing explicit localhost/private IP exceptions for self-hosted setups.
Implements field-level encryption using Fernet (symmetric encryption) with support for Docker secrets as an alternative key source. The encryption key is loaded lazily from environment or file, and the module provides `encrypt()` and `decrypt()` functions that transparently handle string-to-bytes conversions. This is a self-contained, extractable utility for encrypting sensitive configuration values (API keys, passwords) in any Python application that needs credential storage.
Provides a content-type-aware text chunking pipeline. It detects document type (HTML, Markdown, plain text) via file extension and content heuristics, then applies appropriate splitters (e.g., header-based for Markdown/HTML, token-based for plain text). Chunks are further split if they exceed a token budget, and a minimum chunk size filter ensures no null embeddings are generated. This module is reusable in any text processing system that needs to prepare content for embedding or LLM context extraction, independent of the database or AI provider.
Implements source context building for chat and RAG functionalities. It assembles context from sources, notes, and insights, with a token budget estimated via binary search (using a placeholder BPE tokenizer). The budget is distributed among sources with a configurable insight ratio. It marks each source as 'available', 'truncated', 'missing', or 'omitted_budget', and prepends a truncation notice within the budget. This is a reusable pattern for any LLM application that needs to fit variable-length content into a fixed token window while preserving structure and providing transparent truncation metadata.