$ show sgl-project/sglang
SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
The LLM identified 4 files worth highlighting for reuse:
Provides a reusable framework for on-demand CUDA kernel compilation with caching, dependency tracking, and build specification. Includes `cache_once`, `load_jit`, `make_cpp_args`, and `BuildSpec` for parameterized kernel generation. Could be extracted for any project needing runtime CUDA compilation with minimal modification.
Implements a systematic KV cache integrity verification framework using deterministic token oracles, real-KV fingerprinting, and configurable perturbation modes (swap, real-KV used). Includes `CanaryBufferGroup`, `ViolationManager`, and pool patcher adapters. Demonstrates a novel approach to detecting cache corruption in production serving systems.
Defines the abstract `EngineBase` class with methods like `generate`, `flush_cache`, `update_weights_from_tensor`, and `shutdown`. Provides a clean interface for different engine implementations (local, distributed, offline). The pattern is reusable for any inference serving system requiring multiple engine backends.
Implements a `KernelRegistry` with `register_kernel` and `KernelSpec` for metadata-driven kernel selection and fallback. Allows backend-agnostic kernel dispatch by operator name. The pattern is reusable for any system needing extensible operator registration with hardware-specific implementations.