vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
LLaMA-Factory is a production-grade, unified framework for fine-tuning, evaluating, and serving 100+ large language and multimodal models. It provides a CLI (`llamafactory-cli`), a Gradio web UI (LLaMA Board), and an OpenAI-compatible API server. The architecture is monolithic but highly modular, built on Hugging Face Transformers/TRL/PEFT with optional integration of vLLM, SGLang, DeepSpeed, FSDP, Megatron Bridge, and hardware-specific backends (CUDA, ROCm, NPU, XPU). Training pipelines support full-parameter, LoRA, QLoRA, freeze-tuning, and alignment methods (DPO, PPO, KTO, etc.) with extensive data processing (multimodal, packing, streaming, template-based formatting). The codebase features a plugin system (v1) for kernels, quantization, distributed backends, and batching strategies, though it maintains a parallel legacy (v0) path. Reusable components include dataset processors, chat template hierarchy, multimodal plugin architecture, custom trainer callbacks, and batching strategy plugins. While the framework is actively maintained and well-tested, it suffers from some technical drift due to a dual-codebase structure, hardcoded model registries, version pinning, and fragile monkey-patching in distributed integrations.
SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
Tutel is a high-performance library for Mixture-of-Experts (MoE) in PyTorch, focused on both training and inference of large-scale sparsely-gated models. The codebase is organized into distinct layers: the core `MOELayer` configuration (gating, experts, dispatch), custom NCCL-based distributed communication primitives (`net.py`), JIT-compiled CUDA/ROCm kernels for routing and sparse operations, and utility modules for checkpoint resharding across different distributed topologies. Heavily exploited features include low-precision quantized inference (FP4, FP8, block quantization), fused Expert comunication via all-to-all collectives, and a experimental auto-paritioning solver for adaptive parallelism. The repository is dominated by software engineering artifacts (Python library design, C++/CUDA/ROCm extensions and binaries, structured examples) rather than research notebooks or analytic pipeline, but its origins include published papers (MLSys 2023) and active development for contemporary models such as DeepSeek, Qwen3, and Kimi.
fastllm is a high-performance LLM inference engine written in C++ with CUDA/CPU backends, designed for local and server deployment of large language models. In production, it loads models from HuggingFace, GGUF, and custom formats, then executes inference with advanced features: quantization (FP8, INT4, NVFP4, GGUF), speculative decoding, multi-token prediction (MTP), Mixture-of-Experts (MoE), paged KV cache, tensor parallelism, NUMA-aware execution, and disk offloading. It serves models through multiple interfaces: a C++ CLI, a Python CLI (ftllm), an OpenAI-compatible HTTP API server (Chat Completions, Responses, Anthropic Messages), a WebUI, a desktop launcher with agent integrations (DeepSeek Harness, OpenCode, Codex, Claude Code), and an Android app. The system also includes model conversion/export tools, a launcher with hardware auto-recommendation, and a plugin system for UI customization.
This repository provides a minimal, from-scratch implementation of a Vision-Language Model (VLM) called MiniMind-V, designed for educational purposes and low-resource training. It ingests the ALLaVA-4V dataset (Parquet format with image bytes and JSON conversations) for pretraining (1.27M caption pairs) and supervised fine-tuning (2.9M mixed instruct/caption/text). The architecture follows a LLaVA-1.5 style: a frozen SigLIP2 visual encoder (256×256, 64 patch tokens) projects via a 2-layer MLP into a MiniMind LLM (with optional Mixture-of-Experts). Training freezes the visual encoder and trains only the projector and selected LLM layers, achieving ~3 RMB / 2 hours on a single RTX 3090. The codebase includes training scripts (pretrain and SFT), evaluation, a Gradio web demo, and model conversion to HuggingFace Transformers format. While primarily a research/educational project, it demonstrates a clean, modular VLM pipeline that can be adapted for other multimodal tasks.
DeepEP is a high-performance communication library developed by DeepSeek for distributed Mixture-of-Experts (MoE) model training and inference. Its primary function is implementing expert-parallel (EP) all-to-all dispatch and combine operations, routing tokens to the correct GPU holding an expert and gathering the results. It also provides general collective communication primitives (all-reduce, reduce-scatter, all-gather) optimized for multi-node multi-GPU clusters with NVLink and RDMA interconnects. The architecture is a hybrid Python/C++/CUDA library. The Python layer (deep_ep) provides user-facing buffer classes (EPBuffer, BucketBuffer, EngramBuffer, PPBuffer) and orchestration utilities. The C++/CUDA layer (csrc/, deep_ep/include/) contains JIT-compiled kernels that leverage cutting-edge NVIDIA Hopper/Blackwell features (TMA, mbarriers, NCCL GIN) for maximum performance. The library is designed to overlap communication with computation through deferred epilogue patterns. The codebase exhibits exceptional craftsmanship in its low-level CUDA kernels, particularly the ptx.cuh utility layer and the distributed barrier implementations. The Python utilities (EventOverlap, weak_lru, EnvOverride, BufferAllocator) are well-isolated, elegant patterns that are genuinely reusable in other GPU or distributed computing projects. However, the library has a high technical drift risk due to its deep reliance on experimental and rapidly evolving NVIDIA APIs (NCCL device-side, specific CUDA architecture features). It is tightly coupled to a specific hardware stack (Hopper+ GPUs, NVLink, RDMA) and software versions (CUDA 13.1+, NCCL 2.32.3+). While actively maintained, its specialized nature and complex build system limit its applicability outside of large-scale MoE training on cutting-edge NVIDIA clusters.
Time-MoE is a library/framework for training and deploying time series foundation models using a Mixture-of-Experts (MoE) architecture. It provides tools for training the TimeMoE model from scratch or fine-tuning pretrained versions, evaluating on benchmark datasets, and generating forecasts. The codebase is organized into modules for models, datasets, training utilities, and I/O operations, with a clear separation of concerns. Key features include efficient handling of large sequential datasets via sharded binary files, a custom MoE layer with load balancing loss, and integration with Hugging Face Transformers for model management and generation.