$ show turboderp-org/exllamav3
ExLlamaV3 is a high-performance inference library for running large language models on consumer GPUs, with a focus on flexible quantization (EXL3 trellis format) and parallel inference. The architecture is a modular Python library with a compiled C++/CUDA extension providing custom kernels for attention (paged, MLA, DSA, recurrent), MoE routing, quantization, and sampling. Key entry points include the `Model`, `Config`, `Tokenizer`, `Cache`, and `Generator` classes, as well as CLI tools for model conversion (`convert.py`), quantization optimization (`sc_optimize.py`), and evaluation (`eval/`). Data flows: model weights are loaded from safetensors, quantized offline, then at inference time the library manages paged KV caches, supports tensor parallelism across GPUs, and provides streaming generation with filters (Formatron, LLGuidance) and custom samplers. External dependencies include PyTorch, safetensors, tokenizers, Triton, and optional packages like xformers and llguidance. The codebase demonstrates strong engineering with reusable module abstractions, a comprehensive attention dispatch system, and a sophisticated quantization pipeline, though it carries some technical drift from unmaintained dependencies and a large surface of architecture-specific files.
The LLM identified 5 files worth highlighting for reuse:
Central attention dispatch system that selects among Triton paged/varlen/decode/prefill kernels, xformers CUTLASS, torch SDPA fallbacks, and quantized-cache-aware backends. Uses an ordered list of candidate functions returning None on incompatible args, with stale dispatch hints that self-correct. This pattern is reusable for any system needing to select among multiple GPU kernel implementations based on runtime parameters (head dim, cache type, graph mode).
Paged KV cache abstraction with page allocation, copy, and layer-type dispatch. Supports fp16, quantized, MLA, DSA, and recurrent cache variants via a common `CacheLayer` base class. Manages page tables, block tables, and dynamic growth along the sequence dimension. The pattern of a cache layer registry and per-layer-type allocation is reusable for any transformer inference engine needing flexible cache backends.
Base `Filter` class for constrained text generation, with methods `accept_token`, `get_next_logit_mask`, and `is_completed`. Supports journaling for rewindable generation (rollback to previous state). Concrete implementations include `LLGuidanceFilter` and `FormatronFilter`. The filter/journal pattern is reusable for any system needing grammar-guided or schema-constrained output from an LLM.
Provides `get_hadamard()` function that constructs Hadamard matrices of arbitrary sizes using Sylvester construction (power-of-two) and Paley construction (prime-based). Caches results and supports loading precomputed matrices from bundled text files. This is a self-contained utility for any quantization or signal-processing pipeline that requires orthogonal transforms.
Base `Config` class with `read_cfg(type, key, default)` using a `no_default` sentinel to detect missing keys, `assert_cfg` for required fields, and `read_rope_settings_default` for standardized RoPE parameter extraction from HF-style JSON configs. This pattern is reusable for any project that needs robust, type-safe configuration parsing from nested dictionaries.