$ show turboderp-org/exllamav2
ExLlamaV2 is a high-performance inference library for running quantized large language models (LLMs) on consumer GPUs, supporting GPTQ and the custom EXL2 quantization formats. It provides a Python library with custom CUDA/C++ extensions for efficient attention, MLP, sampling, and KV cache management, along with advanced features like dynamic batching, paged attention, speculative decoding, LoRA, tensor parallelism, and multimodal (image/video) support. The library includes a WebSocket server for remote inference and a suite of example scripts demonstrating various use cases. While the codebase exhibits strong engineering with clean abstractions (e.g., module system, filter pattern, multi-backend attention) and highly optimized CUDA kernels, the repository has been archived in favor of ExLlamaV3, meaning no further updates or bug fixes are expected. This makes it a valuable reference for understanding quantized LLM inference internals but risky for new projects requiring ongoing support.
The LLM identified 6 files worth highlighting for reuse:
ExLlamaV2Sampler.Settings is a well-designed dataclass for configuring LLM sampling parameters (temperature, top-k, top-p, min-p, etc.). It includes utility methods like greedy() for deterministic generation, and bias/allow/disallow helpers for logit manipulation. This component is self-contained and could be extracted into any LLM inference project requiring a flexible sampling configuration.
ExLlamaV2Filter is an abstract base class implementing a filter pattern for manipulating logits during generation. It defines begin(), feed(), and next() lifecycle methods, supports background workers, and allows chaining. This pattern is reusable for any token generation system that needs to enforce constraints (e.g., banned strings, JSON schema) without tight coupling to the generator.
SeqTensor is a dynamic tensor container that grows capacity in page-sized increments, useful for managing variable-length sequences in batched inference. It provides efficient append, truncate, and indexing operations. This component is isolated and could be reused in any sequence processing pipeline (e.g., NLP, time series) that needs dynamic memory allocation.
TPContext defines the split configuration for tensor parallelism across multiple GPUs, including memory allocation helpers and broadcast/gather primitives. It is used by linear, attention, and MLP layers to distribute weights. This component is self-contained and could be extracted for any multi-GPU inference framework requiring a clean abstraction for model parallelism.
AdaptiveGPTQ implements the GPTQ quantization algorithm with Hessian reuse, inverse computation, and dampening. It is modular and can be applied to any linear layer. This component is a reference implementation of a state-of-the-art quantization method and could be reused in other quantization pipelines or research projects.
ExLlamaV2Module is a base class for all model submodules (embedding, linear, attention, MLP, etc.) providing a consistent interface for weight loading, device management, footprint calculation, and submodule registration. This pattern is reusable for any deep learning library that needs a structured way to manage model components and their lifecycles.