vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs). It implements PagedAttention for near-zero waste KV cache management, continuous batching, chunked prefill, prefix caching, CUDA/HIP graphs, speculative decoding, and torch.compile. The system serves OpenAI-compatible APIs (chat completions, completions, embeddings, rerank, speech-to-text), Anthropic Messages API, Cohere API, and custom pooling/classification endpoints. It supports 200+ model architectures including dense, MoE, hybrid SSM/attention, and multimodal models (vision, audio, video). The architecture is a hybrid Python/Rust system: a Python core handles model execution, scheduling, and kernel dispatch, while a Rust frontend provides a high-performance HTTP/gRPC server with middleware for auth, metrics, and load management. Key entry points include the `vllm` CLI (serve, bench, launch), the `LLM` class for offline inference, and the OpenAI-compatible API server. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, CUTLASS, Triton, NCCL/RCCL, and various quantization libraries. The project is production-grade with extensive CI/CD, Docker images, Helm charts, and deployment integrations for Kubernetes, Ray, SageMaker, and more.
SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
FastChat is a production-grade distributed serving platform for large language models that powers Chatbot Arena (lmarena.ai), handling 10M+ requests across 70+ models. The architecture centers on a controller that manages model worker registration, heartbeats, and request dispatching; model workers host models via multiple backends (HuggingFace transformers, vLLM, LightLLM, MLX, ExLlamaV2, xFasterTransformer, GPTQ, AWQ, dash-infer); an OpenAI-compatible RESTful API server routes chat/completion/embedding requests; and Gradio web UIs provide single-model chat and side-by-side battle interfaces. The codebase includes training pipelines for Vicuna/FastChat-T5 with FSDP/DeepSpeed/LoRA/QLoRA support, an MT-bench evaluation framework with LLM-as-a-judge (GPT-4), and data processing utilities for ShareGPT conversation cleaning. Key strengths: clean model adapter pattern for adding new architectures, extensible conversation template system, robust streaming API implementation, and well-integrated quantization backends. The system demonstrates non-obvious engineering in distributed inference coordination, attention optimization monkey patches, and multi-backend model loading abstraction.
Capybara is a unified visual creation model that provides inference capabilities for text-to-video, text-to-image, instruction-based video-to-video, and instruction-based image-to-image tasks. The system features a modular architecture with separate components for model handling, pipeline orchestration, and utility functions. Key entry points include the main inference script, ComfyUI custom nodes, and batch/single inference scripts. The system supports distributed multi-GPU inference, prompt rewriting using Qwen3-VL, and FP8 quantization for reduced memory usage.
exo is a distributed inference system for running large language models (LLMs) and image generation models across clusters of devices, primarily targeting Apple Silicon hardware via the MLX framework. It enables running models larger than a single device's memory by sharding them across multiple nodes using pipeline and tensor parallelism. The system provides an OpenAI-compatible API, a web dashboard, and a native macOS application. The architecture is event-sourced, with a master node managing state and workers executing inference tasks. Communication is handled by a custom Rust networking layer built on Zenoh, supporting peer discovery and RDMA over Thunderbolt. The codebase demonstrates strong engineering in several areas, including immutable state models (FrozenModel), typed pub/sub routing, and sophisticated KV-cache management for LLM inference. While the core architecture is sound, the project carries significant technical debt through custom forks of critical dependencies (zenoh, mlx-lm), extensive monkey-patching to work around upstream library limitations, and a highly complex Nix-based build system. The codebase is tightly coupled to the Apple Silicon ecosystem, limiting portability but enabling deep optimization. It is a high-quality, ambitious project under active development, but its complexity and ecosystem coupling present challenges for long-term maintainability and external contribution.
Xinference is a production-grade model serving platform that provides OpenAI-compatible REST APIs for a wide range of AI models including LLMs, embeddings, rerank, image generation, video, audio/TTS, and world models. It supports multiple inference backends (vLLM, SGLang, transformers, MLX, llama.cpp) and can be deployed as a single process or a distributed cluster using an actor-based supervisor/worker architecture (xoscar). The system includes a web UI for model management, authentication/authorization with OAuth2 scopes, token-aware routing for load balancing, and monitoring capabilities. It bundles many third-party model implementations for turnkey deployment, which reduces setup friction but introduces maintenance overhead from vendored code. The architecture is a hybrid of a monolithic FastAPI server and a distributed actor system. Key entry points are CLI commands (xinference, xinference-supervisor, xinference-worker) and the REST API. The frontend is a Next.js application statically exported and served by the backend. The codebase demonstrates strong patterns in modular route registration, plugin-style model registries, and streaming response handling, but also carries significant technical debt from heavy vendoring, duplicated code across third-party integrations, and version-sensitive patches to upstream libraries like vLLM.
dsperse is a distributed Zero-Knowledge Machine Learning (zkML) toolkit that decomposes ONNX neural network models into circuit-compatible segments for verifiable AI inference. The core Rust library provides model slicing, circuit compilation (using JSTprove backend), witness generation, and proof verification. It exposes both a CLI interface and Python bindings via PyO3/Maturin for integration into ML pipelines. The architecture follows a clear separation: the slicer module handles ONNX graph decomposition and tiling strategies (channel split, dimension split, spatial tiling), the pipeline orchestrates execution with memory-efficient tensor caching, and the backend trait enables pluggable ZK proving systems. Content-addressed storage with SHA256 hashing ensures artifact integrity across distributed execution. Notable constraints: the license explicitly restricts operational use without written permission from the IP owner, and the codebase relies on nightly Rust toolchain with specific git dependencies (tract-onnx, jstprove_circuits) that create version coupling risks. Security advisories are partially ignored in the audit configuration.