ExLlamaV2 is a high-performance inference library for running quantized large language models (LLMs) on consumer GPUs, supporting GPTQ and the custom EXL2 quantization formats. It provides a Python library with custom CUDA/C++ extensions for efficient attention, MLP, sampling, and KV cache management, along with advanced features like dynamic batching, paged attention, speculative decoding, LoRA, tensor parallelism, and multimodal (image/video) support. The library includes a WebSocket server for remote inference and a suite of example scripts demonstrating various use cases. While the codebase exhibits strong engineering with clean abstractions (e.g., module system, filter pattern, multi-backend attention) and highly optimized CUDA kernels, the repository has been archived in favor of ExLlamaV3, meaning no further updates or bug fixes are expected. This makes it a valuable reference for understanding quantized LLM inference internals but risky for new projects requiring ongoing support.

ExLlamaV3 is a high-performance inference library for running large language models on consumer GPUs, with a focus on flexible quantization (EXL3 trellis format) and parallel inference. The architecture is a modular Python library with a compiled C++/CUDA extension providing custom kernels for attention (paged, MLA, DSA, recurrent), MoE routing, quantization, and sampling. Key entry points include the `Model`, `Config`, `Tokenizer`, `Cache`, and `Generator` classes, as well as CLI tools for model conversion (`convert.py`), quantization optimization (`sc_optimize.py`), and evaluation (`eval/`). Data flows: model weights are loaded from safetensors, quantized offline, then at inference time the library manages paged KV caches, supports tensor parallelism across GPUs, and provides streaming generation with filters (Formatron, LLGuidance) and custom samplers. External dependencies include PyTorch, safetensors, tokenizers, Triton, and optional packages like xformers and llguidance. The codebase demonstrates strong engineering with reusable module abstractions, a comprehensive attention dispatch system, and a sophisticated quantization pipeline, though it carries some technical drift from unmaintained dependencies and a large surface of architecture-specific files.