$ account cksac
TurboQuant Model is a Python library and CLI tool for near-lossless LLM weight quantization, implementing the TurboQuant algorithm from Zandieh et al. (2025). It provides a drop-in replacement for nn.Linear (TurboQuantLinear) that stores weights in packed 4-bit indices and dequantizes on-the-fly during matrix multiplication using fused GPU kernels (CuTile, Triton, Metal). The library supports single-pass 4-bit, multi-pass residual (e.g., 4+4), polar decomposition, and hash-based compression, achieving 4x compression vs bf16 with KLD as low as 0.002 nats. It includes a CLI for quantizing, evaluating, generating, and benchmarking HuggingFace models, with CPU offload for residual passes and advanced techniques like norm compression and entropy coding. The codebase is well-structured with comprehensive tests and benchmarks, though it targets cutting-edge CUDA 13.1+ and may have limited hardware compatibility.