$ show tobi/qmd
QMD (Quick Markdown Search) is a local-first, on-device hybrid search engine for markdown files, built as a CLI tool, a TypeScript SDK, and an MCP (Model Context Protocol) server. It indexes markdown documents and supports BM25 full-text search via SQLite FTS5, vector semantic search via sqlite-vec, LLM-based query expansion (using a fine-tuned Qwen3-1.7B model), and cross-encoder reranking. The architecture is a monolithic TypeScript module with clear separation of concerns: a core store layer handling indexing and retrieval, a config-driven collection system with YAML-based index definitions, a trust-gated security model for project-local configurations, and output formatters supporting JSON, CSV, Markdown, and XML. The project also includes a fine-tuning pipeline (Python-based SFT, GRPO) to train the query expansion model, with curated training data and evaluation benchmarks. The repository has a hybrid character: while its primary output is a production-ready application (CLI + MCP server), it contains a substantial data-research component in the `finetune/` directory with training scripts, dataset generation, reward functions, and model evaluation harnesses. The fine-tuning pipeline is tightly coupled to the main application's query expansion feature, making this more than a simple add-on – it is integral to the search quality. The code demonstrates strong engineering practices: content-addressable storage for deduplication, AST-aware chunking for code files, reciprocal rank fusion for multi-signal search, and a cross-runtime SQLite abstraction that runs on both Node.js and Bun. The project is actively maintained with CI/CD on GitHub Actions, Nix flake builds, and npm publishing. Key entry points include `src/cli/qmd.ts` (CLI dispatcher), `src/store.ts` (core data layer), `src/mcp/server.ts` (MCP server), and `finetune/train.py` (training pipeline). The system is designed for personal knowledge management and AI agent integration, offering a `qmd search` command for quick lookups and an MCP server exposing tools like `query`, `get`, and `multi_get` for agent consumption. Security is a first-class concern: project-local configs that declare hooks or out-of-project paths require explicit trust approval via a digest-based store, and the HTTP MCP server includes DNS-rebinding protection.
The LLM identified 5 files worth highlighting for reuse:
Cross-runtime SQLite abstraction that unifies `bun:sqlite` and `better-sqlite3` behind a common interface. Handles Homebrew SQLite path resolution on macOS and dynamically loads the sqlite-vec extension. This module is self-contained and can be extracted to provide a portable SQLite layer for any Node/Bun project that needs to abstract away runtime differences.
Digest-based trust store for gating project-local configs that declare sensitive features (hooks, out-of-project paths, custom model URIs). Approvals are recorded per config file and per digest, persisted to a JSON file, and can be revoked. Includes an env-var opt-out for CI. This is a reusable security pattern for any CLI tool that needs to protect users from malicious project-local configurations.
Implements exclusive file-based locking using `fcntl` (POSIX) or advisory file locks to prevent concurrent `qmd embed` processes. Creates a lockfile with the PID and verifies it belongs to a live process. Can be extracted and reused in any tool that needs to serialize resource-intensive background jobs across processes.
AST-aware document chunking for code files (TypeScript, Python, Go, Rust) using tree-sitter grammars. Extracts break points at function boundaries, class definitions, and other structural nodes, falling back to regex-based chunking when grammars are unavailable. This module demonstrates a reusable pattern for language-aware text segmentation that can be adapted to other environments.
DNS-rebinding protection for the HTTP MCP transport. Validates the `Host` and `Origin` headers against a whitelist of allowed origins, preventing attackers from hijacking the MCP server via DNS rebinding. This is a self-contained security middleware that can be reused in any HTTP-based local server that needs to prevent cross-origin attacks.