vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
ExLlamaV3 is a high-performance inference library for running large language models on consumer GPUs, with a focus on flexible quantization (EXL3 trellis format) and parallel inference. The architecture is a modular Python library with a compiled C++/CUDA extension providing custom kernels for attention (paged, MLA, DSA, recurrent), MoE routing, quantization, and sampling. Key entry points include the `Model`, `Config`, `Tokenizer`, `Cache`, and `Generator` classes, as well as CLI tools for model conversion (`convert.py`), quantization optimization (`sc_optimize.py`), and evaluation (`eval/`). Data flows: model weights are loaded from safetensors, quantized offline, then at inference time the library manages paged KV caches, supports tensor parallelism across GPUs, and provides streaming generation with filters (Formatron, LLGuidance) and custom samplers. External dependencies include PyTorch, safetensors, tokenizers, Triton, and optional packages like xformers and llguidance. The codebase demonstrates strong engineering with reusable module abstractions, a comprehensive attention dispatch system, and a sophisticated quantization pipeline, though it carries some technical drift from unmaintained dependencies and a large surface of architecture-specific files.
SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
Mini-SGLang is a lightweight, high-performance inference engine for Large Language Models, designed as a compact reference implementation of the larger SGLang project. It provides an OpenAI-compatible API server, an interactive shell, and a programmatic Python interface. The engine supports advanced optimizations including tensor parallelism (via PyNCCL or NCCL), chunked prefill, radix cache for KV reuse, CUDA graph capture for reduced launch overhead, and multiple attention backends (FlashAttention, FlashInfer, TRT-LLM). It uses a distributed architecture with separate processes for the API server, tokenizer/detokenizer, and per-GPU scheduler workers, communicating via ZeroMQ for control messages and NCCL for tensor data. The codebase is ~5,000 lines of Python with custom CUDA kernels JIT-compiled via TVM-FFI, making it both performant and educational.
This repository is a high-performance computing library optimized for large language model (LLM) inference and training on NVIDIA GPUs. It provides accelerated implementations of core operations including attention mechanisms, GEMM, fused Mixture of Experts (MoE), all-reduce with normalization, sampling, and rotary position embeddings. The architecture combines C++/CUDA kernels for performance-critical operations with Python bindings via PyTorch, enabling easy integration into deep learning frameworks. The library emphasizes kernel fusion, efficient memory access patterns, and specialized optimizations for modern GPU architectures (particularly Hopper and Blackwell).
NanoFlow is a high-throughput LLM serving framework designed to optimize inference through intra-device parallelism and efficient resource utilization. It employs nano-batching (splitting requests at operation granularity) and asynchronous CPU scheduling to overlap compute, memory, and network operations within a single device. The framework supports tensor parallelism for multi-GPU deployment and includes advanced KV-cache management strategies, including offloading to SSDs to avoid recomputation. It is evaluated on models such as Llama2, Llama3, and Qwen2, demonstrating significant throughput improvements over baselines like TensorRT-LLM. The architecture is modular, comprising core components for execution and worker management, model-specific implementations, operation kernels with multiple backends (Torch, CUDA, etc.), KV-cache mechanisms, and utility functions for profiling and resource management. Key entry points include `run_llama3.py` for launching the service and various test scripts for multi-GPU scenarios. Data flows involve request processing through transformer layers, with overlapping of computation and communication via streams and asynchronous execution, managed by a topological executor. The framework relies on a complex dependency stack including NVIDIA CUDA, NCCL, CUTLASS, FlashInfer, MSCCL++, and Gurobi, which introduces maintenance challenges. However, it demonstrates strong engineering practices through reusable components such as the IOWrapper for tensor dependency management, a cross-platform profiler, and a linear programming-based buffer allocator. While active development continues, the repository faces challenges in dependency management and platform fragmentation, indicating a need for careful version control to maintain stability.
fastllm is a high-performance LLM inference engine written in C++ with CUDA/CPU backends, designed for local and server deployment of large language models. In production, it loads models from HuggingFace, GGUF, and custom formats, then executes inference with advanced features: quantization (FP8, INT4, NVFP4, GGUF), speculative decoding, multi-token prediction (MTP), Mixture-of-Experts (MoE), paged KV cache, tensor parallelism, NUMA-aware execution, and disk offloading. It serves models through multiple interfaces: a C++ CLI, a Python CLI (ftllm), an OpenAI-compatible HTTP API server (Chat Completions, Responses, Anthropic Messages), a WebUI, a desktop launcher with agent integrations (DeepSeek Harness, OpenCode, Codex, Claude Code), and an Android app. The system also includes model conversion/export tools, a launcher with hardware auto-recommendation, and a plugin system for UI customization.
NVIDIA's official CUDA Samples repository is a large catalog of self-contained, compilable example programs demonstrating the CUDA platform across C++ and Python. The C++ collection spans hundreds of samples organized into ten categories: fundamentals (vector add, matrix multiply), utilities (deviceQuery), concepts and techniques (reduction, scan, convolution, histograms), feature showcases (CUDA graphs, cooperative groups, Tensor Core GEMMs, dynamic parallelism, NVRTC JIT), CUDA libraries (cuFFT, cuRAND, cuBLAS, cuSOLVER, CUB), domain-specific applications (Black-Scholes, N-body, optical flow, fluid simulation, volume rendering, Monte Carlo), performance studies, libNVVM/PTX generation, platform-specific Tegra/embedded examples, and CUDA Tile C++ microbenchmarks. A parallel Python collection uses the modern cuda.core API with CuPy/NumPy interop and framework integrations (PyTorch, TensorFlow, RAPIDS). Each sample is typically a standalone CLI or interactive graphics program with its own main() entry point; most include a CPU 'gold' reference implementation and automated pass/fail verification, making the collection both educational and regression-tested.
NVIDIA Apex is a PyTorch extension library that provides high-performance, fused CUDA kernels for mixed-precision and distributed deep learning training. In production, it serves as a drop-in acceleration layer for PyTorch models, offering optimized implementations of common operations such as layer normalization, group normalization, fused optimizers (Adam, LAMB, SGD), and structured sparsity (2:4 pruning). The library is not an end-user application but a collection of reusable components that integrate seamlessly with PyTorch's autograd and distributed data-parallel frameworks. The architecture is a hybrid Python/C++/CUDA stack: Python modules expose user-facing APIs (e.g., `apex.optimizers.FusedAdam`, `apex.normalization.FusedLayerNorm`), while the heavy lifting is done by custom CUDA kernels compiled as PyTorch C++ extensions. Key entry points include the `apex.optimizers`, `apex.normalization`, and `apex.contrib` subpackages, which contain both stable and experimental features. Data flows are typical of PyTorch training loops: forward/backward passes invoke autograd Functions that dispatch to fused CUDA kernels, and distributed training utilities (e.g., `DistributedFusedAdam`) overlap gradient synchronization with computation. External dependencies include PyTorch, CUDA, cuDNN, and NCCL. The library also includes experimental modules for TorchInductor multi-stream scheduling, OpenFold-specific Triton kernels, and cuDNN frontend-based operations. While the core components are well-maintained, some contrib modules show signs of technical drift (duplicate optimizer implementations, deprecated API usage, hardcoded shape constraints). Overall, Apex is a mature, high-quality library that has been widely adopted in the PyTorch ecosystem for accelerating large-scale model training.
BitNet is a high-performance inference framework for 1-bit large language models, specifically the BitNet b1.58 ternary architecture. It provides CPU (x86 with AVX2, ARM with NEON) and GPU (NVIDIA CUDA) backends with custom quantization formats (I2_S, TL1, TL2) that pack ternary weights and use efficient look-up tables and integer matrix multiplication (int8×int2) to achieve significant speedups over full-precision models (1.37–6.17× on CPU). The system is built on a fork of llama.cpp, adding custom GEMM kernels and a Python utility suite for model conversion, embedding quantization, benchmark, and deployment. It includes a CLI interactive mode and an HTTP server for production serving. The repository contains extensive performance benchmarks and configuration tools, making it a production-ready solution for deploying 1-bit LLMs on commodity hardware.
ChatRWKV is a collection of inference tools, demos, and a pip package for the RWKV language model family (v4-v7), a 100% RNN architecture that rivals transformers in performance. The repository provides interactive chat scripts (chat.py, v2/chat.py), API demos (API_DEMO_CHAT.py), model conversion utilities, state merging, music generation (MIDI/ABC), and benchmarking scripts. The core RWKV model class (rwkv_pip_package/src/rwkv/model.py) supports flexible deployment strategies (CPU/GPU, fp16/fp32/bf16, int8 quantization, streaming) and includes custom CUDA kernels for efficient WKV operations. The tokenizer module (tokenizer/rwkv_tokenizer.py) offers a fast trie-based byte-level tokenizer alongside a reference implementation. While the repository is primarily a research/demo project with many ad-hoc scripts and hardcoded paths, the pip package and core components are well-crafted and demonstrate significant engineering skill in GPU optimization and model inference. The codebase is actively maintained and supports the latest RWKV-7 architecture.
DeepEP is a high-performance communication library developed by DeepSeek for distributed Mixture-of-Experts (MoE) model training and inference. Its primary function is implementing expert-parallel (EP) all-to-all dispatch and combine operations, routing tokens to the correct GPU holding an expert and gathering the results. It also provides general collective communication primitives (all-reduce, reduce-scatter, all-gather) optimized for multi-node multi-GPU clusters with NVLink and RDMA interconnects. The architecture is a hybrid Python/C++/CUDA library. The Python layer (deep_ep) provides user-facing buffer classes (EPBuffer, BucketBuffer, EngramBuffer, PPBuffer) and orchestration utilities. The C++/CUDA layer (csrc/, deep_ep/include/) contains JIT-compiled kernels that leverage cutting-edge NVIDIA Hopper/Blackwell features (TMA, mbarriers, NCCL GIN) for maximum performance. The library is designed to overlap communication with computation through deferred epilogue patterns. The codebase exhibits exceptional craftsmanship in its low-level CUDA kernels, particularly the ptx.cuh utility layer and the distributed barrier implementations. The Python utilities (EventOverlap, weak_lru, EnvOverride, BufferAllocator) are well-isolated, elegant patterns that are genuinely reusable in other GPU or distributed computing projects. However, the library has a high technical drift risk due to its deep reliance on experimental and rapidly evolving NVIDIA APIs (NCCL device-side, specific CUDA architecture features). It is tightly coupled to a specific hardware stack (Hopper+ GPUs, NVLink, RDMA) and software versions (CUDA 13.1+, NCCL 2.32.3+). While actively maintained, its specialized nature and complex build system limit its applicability outside of large-scale MoE training on cutting-edge NVIDIA clusters.