AIBrix is a Kubernetes-native control plane and intelligent gateway for serving production LLM workloads. The repository implements two intertwined planes: a data plane composed of Envoy ext_proc gateway plugins (cmd/plugins, pkg/plugins/gateway) that intercept OpenAI-compatible inference requests and route them to vLLM, SGLang, or TensorRT-LLM replicas using algorithms such as least-load, prefix-cache-aware routing, SLO queueing, token-load tracking, and prefill/decode disaggregation; and a control plane of Kubernetes controllers (cmd/controllers) that reconcile custom resources for autoscaling (PodAutoscaler), LoRA adapters, model warmup/claims, KV caches, RayCluster fleets, and StormService/RoleSet orchestration. Supporting runtime code includes a Python metadata and batch-job service, a distributed KV-cache package, a GPU optimizer, and an OpenAI-compatible frontend.
Mini-SGLang is a lightweight, high-performance inference engine for Large Language Models, designed as a compact reference implementation of the larger SGLang project. It provides an OpenAI-compatible API server, an interactive shell, and a programmatic Python interface. The engine supports advanced optimizations including tensor parallelism (via PyNCCL or NCCL), chunked prefill, radix cache for KV reuse, CUDA graph capture for reduced launch overhead, and multiple attention backends (FlashAttention, FlashInfer, TRT-LLM). It uses a distributed architecture with separate processes for the API server, tokenizer/detokenizer, and per-GPU scheduler workers, communicating via ZeroMQ for control messages and NCCL for tensor data. The codebase is ~5,000 lines of Python with custom CUDA kernels JIT-compiled via TVM-FFI, making it both performant and educational.
NanoFlow is a high-throughput LLM serving framework designed to optimize inference through intra-device parallelism and efficient resource utilization. It employs nano-batching (splitting requests at operation granularity) and asynchronous CPU scheduling to overlap compute, memory, and network operations within a single device. The framework supports tensor parallelism for multi-GPU deployment and includes advanced KV-cache management strategies, including offloading to SSDs to avoid recomputation. It is evaluated on models such as Llama2, Llama3, and Qwen2, demonstrating significant throughput improvements over baselines like TensorRT-LLM. The architecture is modular, comprising core components for execution and worker management, model-specific implementations, operation kernels with multiple backends (Torch, CUDA, etc.), KV-cache mechanisms, and utility functions for profiling and resource management. Key entry points include `run_llama3.py` for launching the service and various test scripts for multi-GPU scenarios. Data flows involve request processing through transformer layers, with overlapping of computation and communication via streams and asynchronous execution, managed by a topological executor. The framework relies on a complex dependency stack including NVIDIA CUDA, NCCL, CUTLASS, FlashInfer, MSCCL++, and Gurobi, which introduces maintenance challenges. However, it demonstrates strong engineering practices through reusable components such as the IOWrapper for tensor dependency management, a cross-platform profiler, and a linear programming-based buffer allocator. While active development continues, the repository faces challenges in dependency management and platform fragmentation, indicating a need for careful version control to maintain stability.
This is a research codebase for the Geometric Context Transformer (GCT), a streaming monocular 3D reconstruction model accepted at ECCV 2026. The system processes RGB video frames in real-time (targeting ~20 FPS) to predict camera poses, depth maps, and 3D point clouds using a vision transformer with a novel paged KV-cache causal attention mechanism. It includes a full inference pipeline, interactive and offline renderers (based on Viser and Open3D/CUDA voxelization with octree LOD), and a comprehensive benchmark framework supporting multiple datasets (KTH, Oxford Spires, TUM, 7-Scenes, ETH3D, etc.) with trajectory, depth, and point cloud metrics. The codebase is predominantly application-oriented—providing a deployable model and evaluation harness—but incorporates significant data engineering components such as dataset preprocessing (e.g., Oxford Spires fisheye-to-pinhole conversion with CUDA visibility filtering), standardized BSS artifact storage, and reproducible conda-based environment setup. The core model architecture demonstrates advanced engineering practices: two-stream token layouts, sliding-window KV cache eviction with special-token preservation, 3D RoPE, and modular attention backends (FlashInfer, SDPA, custom causal). The benchmark framework is well-architected with pluggable methods and datasets, resumable execution, and static HTML report generation.
This repository is a research prototype for efficient LLM inference, specifically optimizing attention memory management via the vAttention technique (ASPLOS 2025). It contains three primary components: (1) a modified NVIDIA GPU kernel driver (nvidia-vattn-uvm-driver) that enables sub-2MB page sizes for Unified Virtual Memory (UVM), (2) a fork of the Sarathi-Serve LLM serving framework (sarathi-lean) integrated with vAttention, and (3) custom attention kernel implementations (pod_attn, vattention) using CUTLASS and FlashAttention backends. The system supports multiple attention backends (FlashAttention, FlashInfer, paged, variable-length) and includes extensive benchmarking and plotting scripts for reproducibility of research results. Architecturally, the user-space inference stack uses a scheduler (Orca or Sarathi), a cache engine (vLLM-style or vAttention), and model executors with tensor/pipeline parallelism. The vAttention cache engine replaces traditional paging with CUDA virtual memory allocation, decoupling virtual and physical memory for KV-cache. The custom NVIDIA UVM driver patch is required to support fine-grained page sizes. The repository also includes microbenchmarks for attention kernel performance and page size effects, as well as artifact scripts to reproduce figures from the associated paper. While the codebase demonstrates solid engineering for a research project (clean abstractions, registry pattern, modular design), it is explicitly not production-ready. Key limitations include strict version dependencies (PyTorch 2.3.0, CUDA 12.1), reliance on a modified kernel driver, and incomplete feature parity with production systems like vLLM. The inclusion of a fully vendored CUTLASS library and auto-generated NVIDIA driver headers adds significant size but provides reusable GPU primitives. The research scripts are functional but contain hardcoded paths and are tightly coupled to the experimental setup.