$ show efeslab/Nanoflow
NanoFlow is a high-throughput LLM serving framework designed to optimize inference through intra-device parallelism and efficient resource utilization. It employs nano-batching (splitting requests at operation granularity) and asynchronous CPU scheduling to overlap compute, memory, and network operations within a single device. The framework supports tensor parallelism for multi-GPU deployment and includes advanced KV-cache management strategies, including offloading to SSDs to avoid recomputation. It is evaluated on models such as Llama2, Llama3, and Qwen2, demonstrating significant throughput improvements over baselines like TensorRT-LLM. The architecture is modular, comprising core components for execution and worker management, model-specific implementations, operation kernels with multiple backends (Torch, CUDA, etc.), KV-cache mechanisms, and utility functions for profiling and resource management. Key entry points include `run_llama3.py` for launching the service and various test scripts for multi-GPU scenarios. Data flows involve request processing through transformer layers, with overlapping of computation and communication via streams and asynchronous execution, managed by a topological executor. The framework relies on a complex dependency stack including NVIDIA CUDA, NCCL, CUTLASS, FlashInfer, MSCCL++, and Gurobi, which introduces maintenance challenges. However, it demonstrates strong engineering practices through reusable components such as the IOWrapper for tensor dependency management, a cross-platform profiler, and a linear programming-based buffer allocator. While active development continues, the repository faces challenges in dependency management and platform fragmentation, indicating a need for careful version control to maintain stability.
The LLM identified 3 files worth highlighting for reuse:
Defines a unified tensor interface that tracks dependencies, shapes, and device placement. This component is self-contained, demonstrating a novel approach to managing dynamic computation graphs in LLM serving, and can be extracted for use in other projects requiring tensor dependency resolution.
Provides a cross-platform profiling interface supporting NVTX, ROCm, and PyTorch. This small, self-contained utility abstracts platform-specific profiling mechanisms, making it easily reusable in other GPU-accelerated applications for performance analysis.
Implements buffer allocation using linear programming (via Gurobi) to optimize memory usage for tensors. This component showcases non-obvious skill in formulating memory allocation as an optimization problem and, despite its reliance on a commercial solver, presents a reusable pattern for resource-constrained environments.