Witchcraft is a high-performance, local-first semantic search engine built in Rust. It indexes documents using a quantized T5 encoder (Google DeepMind's XTR model) and stores dense and sparse embeddings in SQLite, enabling hybrid retrieval with no external API dependencies or vector database. The project provides multiple interfaces: a CLI (`warp`), a Python package (`witchcraft`), a Node.js native module (`warp`), and an MCP server for AI agent integration. The flagship use case is "pickbrain" — a TUI application that ingests conversation transcripts from Pi, Claude Code, Codex, and Slack, then allows the user to search and resume sessions directly from the terminal. The architecture is modular: the core library (`src/lib.rs`) handles embedding, indexing, and search; `src/embedder.rs` wraps tokenization and T5 inference with sliding-window support; `src/db.rs` manages a SQLite database with WAL, FTS5, and batch operations. Performance-critical operations use custom CPU kernels in `src/fast_ops.rs` and the separate `fbgemm-rs` crate, which provides hand-tuned AVX2/NEON bf16 matrix multiplication and int8 quantized GEMM. Embedding compression via Haar wavelets and rANS entropy coding is provided by `src/packops.rs` and `src/rans64.rs`. The project also includes benchmark tools, a quantization tool, and safety netting for platform-specific quirks (OpenVINO memory leaks, macOS thread priorities). Overall, it is a well-crafted application demonstrating deep systems engineering, though its complexity and tight coupling of some components limit broad reuse.
AutoRAG is a production-grade, privacy-focused personal AI assistant and retrieval-augmented generation (RAG) agent. It provides a local-first search and answer engine that indexes federated datasources (Slack, Discord, KakaoTalk, GitHub, email, Obsidian, RSS, cloud drives, and more) via external CLI connectors, without migrating data. The system exposes a CLI, MCP server, TypeScript API, and a TUI (Pi interactive host), and powers a separate desktop Electron app. Its architecture is modular: a configurable agent (`AutoRAGAgent`) orchestrates multiple retrieval methods (BM25, semantic vectors, hybrid RRF, file-name search, web search), document parsing (PDF, DOCX, HWP, images via OCR), embedding (local Qwen3 model or external APIs), reranking (OpenRouter/Dazzi), and evidence judgment. A peer-to-peer layer (SimpleX) enables policy-controlled file sharing with injection and PII gates. The repository also contains a legacy Python codebase (`legacy/`) focused on automated RAG pipeline evaluation and QA dataset generation, which is in maintenance mode but still provides reusable metrics and pipeline components. This is a hybrid project where the dominant character is a modern TypeScript application, with significant data/research heritage in the legacy Python portion.
This repository is a Docusaurus v3+ plugin that provides offline/local search functionality. It operates in two phases: at build time, it parses the generated HTML pages (docs, blog, static pages) using cheerio, extracts text and metadata, and builds Lunr.js search indexes per documentation version/language. At runtime, it serves a React-based search bar using @algolia/autocomplete-js that fetches the pre-built index JSON and performs client-side searches with Lunr. The plugin supports multiple languages, documentation versions, i18n, and configurable boosting of title/content/tags. It is a well-engineered monorepo with a single plugin package and an example Docusaurus site for testing. The code is TypeScript, has unit and e2e tests, and follows modern development practices (pnpm workspaces, Turborepo, conventional commits).
MaterialSearch is a fully offline, local media search engine that uses CLIP (Chinese-CLIP) to search photos and videos by natural language or image similarity. The repository contains the frontend web application and a desktop GUI launcher; the core search logic (scanning, embedding, matching) lives in a separate, obfuscated pip package (`materialsearch-core`). The frontend is a Vue 3 single-page application built with Element Plus, providing five search modes (text-to-image, image-to-image, text-to-video, image-to-video, text-image similarity), along with file path copying, clipboard image paste, and video clip download. Deployment options include a Docker container (via docker-compose.yml) or a Windows executable bundle. The system is designed to run entirely on the user's machine with no data uploaded to the cloud, making it suitable for privacy-sensitive or air-gapped environments.