$ show sgl-project/mini-sglang
Mini-SGLang is a lightweight, high-performance inference engine for Large Language Models, designed as a compact reference implementation of the larger SGLang project. It provides an OpenAI-compatible API server, an interactive shell, and a programmatic Python interface. The engine supports advanced optimizations including tensor parallelism (via PyNCCL or NCCL), chunked prefill, radix cache for KV reuse, CUDA graph capture for reduced launch overhead, and multiple attention backends (FlashAttention, FlashInfer, TRT-LLM). It uses a distributed architecture with separate processes for the API server, tokenizer/detokenizer, and per-GPU scheduler workers, communicating via ZeroMQ for control messages and NCCL for tensor data. The codebase is ~5,000 lines of Python with custom CUDA kernels JIT-compiled via TVM-FFI, making it both performant and educational.
The LLM identified 1 files worth highlighting for reuse:
Provides generic JIT and AOT compilation helpers for CUDA and C++ kernels using TVM-FFI. The `load_jit` and `load_aot` functions encapsulate the complexity of compiling source code at runtime, managing include paths, and registering exported functions. This component is self-contained (only depends on tvm_ffi and standard library) and can be extracted and reused in any project that needs to compile and call custom CUDA kernels dynamically.