SeekStorm is a high-performance search engine library and multi-tenancy server implemented in Rust. It provides both lexical (BM25F with proximity) and vector (ANN with IVF clustering and scalar/Turbo quantization) search capabilities, supporting real-time indexing, faceted search, spelling correction, query auto-completion, highlighting, and hybrid result fusion via RRF. The architecture consists of a core library (`seekstorm`), a RESTful server (`seekstorm_server`) with embedded web UI, and a Rust client SDK (`seekstorm_client`). The server uses memory-mapped files for persistence, supports per-index configuration of tokenizers, stemmers, stop words, and compression, and enforces multi-tenant isolation via API keys with resource quotas. Search performance is enhanced with SIMD acceleration (AVX2/NEON) for posting list intersection, vector quantization, and distance computations. The codebase is well-tested with extensive integration and consistency tests, and follows an explicit commit model for durability.
This repository is a research codebase for implementing and experimenting with dense retrieval techniques. It provides implementations of multiple advanced retrieval models including DPR, CITADEL, COIL, ColBERT, SPLADE, and their variants, along with data augmentation strategies like DRAGON and DRAMA. The codebase enables researchers to train, evaluate, and compare different retrieval approaches on standard benchmarks like Natural Questions, MS MARCO, and conversational datasets. It features a modular architecture with separate components for data processing, model definitions, training tasks, and evaluation, configured through Hydra for easy experimentation.