vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
SGLang-JAX is a high-performance LLM inference engine optimized for TPU/GPU hardware using JAX. It implements continuous batching, radix tree KV cache management, FlashAttention integration, tensor parallelism, and disaggregated serving (prefill-decode separation). The system provides OpenAI-compatible API endpoints for text generation, chat, embeddings, and tool use, with support for Mixture-of-Experts models, LoRA adaptation, quantization, and speculative decoding. Architecture features modular components: tokenizer manager, scheduler, model executor, memory hierarchy (including HiCache host-device transfer), optimized JAX/Pallas kernels, and LoRA adapter management. Designed for production serving workloads on TPU infrastructure with first-party support for Qwen, Llama, Gemma, DeepSeek, and multimodal models.
SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
This repository provides a Dart FFI binding for llama.cpp, targeting Flutter mobile apps on iOS, Android, and macOS. It wraps the native C API with a modular, isolate-safe architecture: a worker isolate (LlamaEngine) owns the native model and context, while the main isolate communicates via message passing. The binding exposes streaming token generation, chat template rendering, multimodal (vision/audio) support, speculative decoding, context shifting, and session persistence. The codebase is well-structured with clear separation of concerns: FFI bindings, model/context wrappers, sampling, tokenization, and generation logic are each in their own modules. It uses modern Dart features (sealed classes, records, pattern matching) and demonstrates high craftsmanship in error handling, resource management, and cross-isolate communication. The package is a Flutter plugin with native-assets build hooks for Android and Swift Package Manager support for Apple platforms.
This repository is a research codebase for training and evaluating draft models for speculative decoding, supporting three algorithms: DSpark, DFlash, and Eagle3. It provides a full pipeline from data preparation (downloading prompts, regenerating answers with a target model via SGLang, and precomputing a large binary cache of target hidden states) to distributed training with FSDP and BF16, and evaluation on standard LLM benchmarks (GSM8K, MATH500, AIME, HumanEval, etc.). The code is well-structured with reusable components for configuration, distributed utilities, checkpointing, metrics, and custom attention masks. While primarily a research artifact, it exhibits strong engineering practices and several modules that could be extracted for other large-scale ML projects. The repository is hybrid in nature: it contains both application-like training/evaluation scripts and data_research elements such as data pipelines and evaluation datasets.
fastllm is a high-performance LLM inference engine written in C++ with CUDA/CPU backends, designed for local and server deployment of large language models. In production, it loads models from HuggingFace, GGUF, and custom formats, then executes inference with advanced features: quantization (FP8, INT4, NVFP4, GGUF), speculative decoding, multi-token prediction (MTP), Mixture-of-Experts (MoE), paged KV cache, tensor parallelism, NUMA-aware execution, and disk offloading. It serves models through multiple interfaces: a C++ CLI, a Python CLI (ftllm), an OpenAI-compatible HTTP API server (Chat Completions, Responses, Anthropic Messages), a WebUI, a desktop launcher with agent integrations (DeepSeek Harness, OpenCode, Codex, Claude Code), and an Android app. The system also includes model conversion/export tools, a launcher with hardware auto-recommendation, and a plugin system for UI customization.
This repository implements Speculative Speculative Decoding (SSD), a novel LLM inference algorithm that enables parallel execution of draft and target models on separate hardware. The system provides a complete inference engine with tensor parallelism support, optimized kernels (FlashInfer, CUDA graphs), and advanced KV caching techniques. It includes benchmarking tools to compare performance against baselines like SGLang and vLLM, and supports both synchronous and asynchronous speculative decoding modes. The codebase is well-structured with clear separation of concerns between model implementations, engine components, and utility functions.
This repository is a curated collection of learning materials for AI infrastructure, specifically focused on large language model (LLM) inference optimization. It contains detailed markdown documents covering core topics such as vLLM quickstart, PagedAttention, Prefix Caching, Speculative Decoding, Chunked-Prefills, and Prefill-Decode (PD) Disaggregation. Each lesson includes in-depth explanations, diagrams, code examples, and references to relevant papers and talks, making it a comprehensive resource for engineers and researchers seeking to understand and apply state-of-the-art LLM serving techniques. The content is organized as a series of study sessions, with each lesson building on previous ones. The repository does not contain any runnable application code or data pipelines; it is purely educational material. However, the code snippets provided (e.g., Python scripts for vLLM offline and online inference) are directly usable and demonstrate practical usage of the discussed technologies. The documentation is well-structured, with clear learning objectives, prerequisites, and supplementary resources. While the repository itself is not a software application, its value lies in the depth and clarity of its technical explanations. It serves as an excellent reference for anyone deploying or optimizing LLM inference systems, and the content is actively maintained with recent commits and issue tracking. The absence of reusable software components is expected given its educational nature, but the quality of the documentation is high, earning a strong quality score.