$ show cksac/turboquant-model
TurboQuant Model is a Python library and CLI tool for near-lossless LLM weight quantization, implementing the TurboQuant algorithm from Zandieh et al. (2025). It provides a drop-in replacement for nn.Linear (TurboQuantLinear) that stores weights in packed 4-bit indices and dequantizes on-the-fly during matrix multiplication using fused GPU kernels (CuTile, Triton, Metal). The library supports single-pass 4-bit, multi-pass residual (e.g., 4+4), polar decomposition, and hash-based compression, achieving 4x compression vs bf16 with KLD as low as 0.002 nats. It includes a CLI for quantizing, evaluating, generating, and benchmarking HuggingFace models, with CPU offload for residual passes and advanced techniques like norm compression and entropy coding. The codebase is well-structured with comprehensive tests and benchmarks, though it targets cutting-edge CUDA 13.1+ and may have limited hardware compatibility.
The LLM identified 5 files worth highlighting for reuse:
Generates Lloyd-Max codebooks for any bit-width (1-16) using precomputed Gaussian CDF and inverse CDF. The codebook is cached globally and is reusable for any scalar quantization task requiring optimal centroids for a Gaussian-distributed variable. The logic is isolated and can be extracted into a standalone utility.
Provides rotation matrices via QR (Haar) or fast Walsh-Hadamard transform. The hadamard_rotate function uses O(d log d) operations and is a self-contained utility for applying structured rotations to vectors. The module is decoupled from quantization and can be reused in any rotation-based compression or privacy algorithm.
Contains pack_4bit and unpack_4bit functions that pack two 4-bit integers into a single uint8 byte, and vice versa. These are simple, well-tested utilities that can be reused in any project requiring 4-bit storage or serialization. The module also includes the core quantization loop, but the packing functions are standalone.
Implements a range-Asymmetric Numeral Systems (rANS) codec specifically designed for quantized indices with Gaussian-distributed bin probabilities. The codec precomputes decode tables and can operate in GPU shared memory. It is a self-contained, reusable entropy coding module that could be applied to any quantization scheme with a known probability distribution.
Implements multi-pass residual quantization and a merge algorithm (merge_residual_passes, merge_and_requantize) that collapses several residual passes into a single-pass format with shared rotation. This is a non-trivial, reusable pattern for any multi-level quantization scheme, allowing storage and inference efficiency improvements.