SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.

Mini-SGLang is a lightweight, high-performance inference engine for Large Language Models, designed as a compact reference implementation of the larger SGLang project. It provides an OpenAI-compatible API server, an interactive shell, and a programmatic Python interface. The engine supports advanced optimizations including tensor parallelism (via PyNCCL or NCCL), chunked prefill, radix cache for KV reuse, CUDA graph capture for reduced launch overhead, and multiple attention backends (FlashAttention, FlashInfer, TRT-LLM). It uses a distributed architecture with separate processes for the API server, tokenizer/detokenizer, and per-GPU scheduler workers, communicating via ZeroMQ for control messages and NCCL for tensor data. The codebase is ~5,000 lines of Python with custom CUDA kernels JIT-compiled via TVM-FFI, making it both performant and educational.

SGLang-JAX is a high-performance LLM inference engine optimized for TPU/GPU hardware using JAX. It implements continuous batching, radix tree KV cache management, FlashAttention integration, tensor parallelism, and disaggregated serving (prefill-decode separation). The system provides OpenAI-compatible API endpoints for text generation, chat, embeddings, and tool use, with support for Mixture-of-Experts models, LoRA adaptation, quantization, and speculative decoding. Architecture features modular components: tokenizer manager, scheduler, model executor, memory hierarchy (including HiCache host-device transfer), optimized JAX/Pallas kernels, and LoRA adapter management. Designed for production serving workloads on TPU infrastructure with first-party support for Qwen, Llama, Gemma, DeepSeek, and multimodal models.