$ show VexDB-THU/VexDB-Lite
VexDB-Lite is a vector similarity search library providing Approximate Nearest Neighbor (ANN) capabilities as extensions for PostgreSQL, DuckDB, and SQLite databases. The system implements Hierarchical Navigable Small World (HNSW) graphs for efficient similarity search, with optional Product Quantization and RaBitQ compression for memory efficiency. It features hardware-accelerated distance computation using SIMD instructions (SSE, AVX, AVX512, NEON) and provides multiple distance metrics (L2, inner product, cosine). The architecture separates core algorithms from database-specific adapters, enabling consistent functionality across different storage engines. The codebase demonstrates strong engineering practices with reusable components like the SIMD dispatch system, vector buffer manager, and product quantization implementation.
The LLM identified 5 files worth highlighting for reuse:
Provides runtime selection of architecture-optimized distance functions based on CPU capabilities. Implements a clean dispatcher pattern that abstracts SIMD optimizations away from calling code, allowing transparent use of SSE, AVX, AVX512, or NEON implementations. This component could be extracted for any numerical computation requiring hardware acceleration.
Implements a thread-safe LRU cache for vector data with background eviction mechanisms and shared memory support. Designed for database buffer management but applicable to any caching scenario requiring concurrent access and automatic eviction. Demonstrates sophisticated cache design with producer-consumer patterns for background maintenance.
Self-contained Product Quantization implementation with training, encoding, and decoding capabilities. Uses template metaprogramming for flexible configuration of subvector count and bit allocation. Could be used for vector compression in any context outside of similarity search, demonstrating clean separation of quantization logic from storage concerns.
Core HNSW (Hierarchical Navigable Small World) graph algorithm implementation separated from storage/persistence concerns. Provides the fundamental graph construction, insertion, and search logic that can be paired with different storage backends. Demonstrates clean separation of algorithmic concerns from implementation details.
Custom hash table implementation with allocator awareness and policy-based design. Part of the VTL (Versatile Template Library) that provides container abstractions with explicit control over memory allocation and collision resolution strategies. Shows advanced C++ techniques for creating flexible, reusable data structures.