NanoFlow is a high-throughput LLM serving framework designed to optimize inference through intra-device parallelism and efficient resource utilization. It employs nano-batching (splitting requests at operation granularity) and asynchronous CPU scheduling to overlap compute, memory, and network operations within a single device. The framework supports tensor parallelism for multi-GPU deployment and includes advanced KV-cache management strategies, including offloading to SSDs to avoid recomputation. It is evaluated on models such as Llama2, Llama3, and Qwen2, demonstrating significant throughput improvements over baselines like TensorRT-LLM. The architecture is modular, comprising core components for execution and worker management, model-specific implementations, operation kernels with multiple backends (Torch, CUDA, etc.), KV-cache mechanisms, and utility functions for profiling and resource management. Key entry points include `run_llama3.py` for launching the service and various test scripts for multi-GPU scenarios. Data flows involve request processing through transformer layers, with overlapping of computation and communication via streams and asynchronous execution, managed by a topological executor. The framework relies on a complex dependency stack including NVIDIA CUDA, NCCL, CUTLASS, FlashInfer, MSCCL++, and Gurobi, which introduces maintenance challenges. However, it demonstrates strong engineering practices through reusable components such as the IOWrapper for tensor dependency management, a cross-platform profiler, and a linear programming-based buffer allocator. While active development continues, the repository faces challenges in dependency management and platform fragmentation, indicating a need for careful version control to maintain stability.
NVIDIA cuOpt is a GPU-accelerated optimization solver library covering linear programming (LP), mixed-integer programming (MIP), quadratic programming (QP), second-order cone programming (SOCP), and vehicle routing problems (VRP, PDP, TSP). The codebase is predominantly an application: it provides multiple deployment forms—a C++ shared library, Python bindings (Cython), a Java JNI interface, a CLI tool (`cuopt_cli`), a gRPC server for remote solves, and a FastAPI-based HTTP server (`cuopt_server`). It also includes comprehensive CI/CD pipelines, benchmarks, and documentation. However, it exhibits a hybrid nature due to its strong research character: advanced algorithmic innovations (e.g., primal-dual hybrid gradient with adaptive step sizing, novel MIP presolve techniques, lattice propagation, deterministic parallel branch-and-bound, SIMD-accelerated MPS parsing, and diversity-based routing heuristics) are deeply embedded in the core solvers. The system is architected as a layered monolith: common utilities and linear algebra form the base, solver-specific modules (PDLP, dual simplex, barrier, branch-and-bound, routing heuristics) sit above, and language bindings and server endpoints wrap the C++ core. Key data flows include parsing MPS/LP files or building problems programmatically, dispatching to the appropriate GPU solver, and retrieving solutions via C structs, Python objects, or protobuf messages. External dependencies include CUDA, cuSPARSE, RMM, NCCL, gRPC, Protobuf, and optionally PaPILO for presolve. The repository is actively maintained with frequent commits and nightly CI, though the presence of deprecated modules and experimental features indicates moderate technical drift.
This repository is the LLM Inference Handbook, a comprehensive technical documentation site built with Docusaurus. It serves as an educational resource for engineers building, optimizing, and deploying LLM inference systems. The site covers inference fundamentals, model preparation, interaction patterns, optimization techniques (batching, KV cache, prefix caching, speculative decoding), kernel optimization (FlashAttention, GPU architecture), and infrastructure operations. It includes interactive React components such as GPU memory calculators, KV cache simulators, batching visualizers, and GPU execution visualizers that demonstrate concepts through live demos. The codebase is well-structured with TypeScript, CSS modules, a custom design system (DLS), and integration with Mantine UI. While primarily a documentation site, the interactive components are self-contained and could be extracted for reuse in other educational contexts. The handbook is framework-agnostic but has a strong emphasis on the Modular MAX ecosystem.