$ show lm-sys/FastChat
FastChat is a production-grade distributed serving platform for large language models that powers Chatbot Arena (lmarena.ai), handling 10M+ requests across 70+ models. The architecture centers on a controller that manages model worker registration, heartbeats, and request dispatching; model workers host models via multiple backends (HuggingFace transformers, vLLM, LightLLM, MLX, ExLlamaV2, xFasterTransformer, GPTQ, AWQ, dash-infer); an OpenAI-compatible RESTful API server routes chat/completion/embedding requests; and Gradio web UIs provide single-model chat and side-by-side battle interfaces. The codebase includes training pipelines for Vicuna/FastChat-T5 with FSDP/DeepSpeed/LoRA/QLoRA support, an MT-bench evaluation framework with LLM-as-a-judge (GPT-4), and data processing utilities for ShareGPT conversation cleaning. Key strengths: clean model adapter pattern for adding new architectures, extensible conversation template system, robust streaming API implementation, and well-integrated quantization backends. The system demonstrates non-obvious engineering in distributed inference coordination, attention optimization monkey patches, and multi-backend model loading abstraction.
The LLM identified 8 files worth highlighting for reuse:
Central controller managing worker registration, health checks via heartbeats, model listing, and request dispatching with configurable strategies (shortest_queue, lottery). Clean separation from worker implementation enables multiple backend types.
Base class for all model workers providing FastAPI endpoints for generation (streaming/non-streaming), embeddings, token counting, and conversation templates. Uses asyncio semaphores for concurrency limiting and integrates with controller heartbeat system.
Full OpenAI API implementation (chat/completions/embeddings) with proper SSE streaming, Pydantic request/response models, OpenAI-compatible error codes, API key auth, and model routing via controller. Handles both OpenAI and custom API protocols.
Extensible conversation template system with register_conv_template() for adding new model formats. Handles system prompts, role mappings, separator styles, stop tokens, and vision/multimodal message formatting. Used across CLI, web UI, API server, and workers.
Model loading abstraction supporting HF transformers, GPTQ, AWQ, ExLlamaV2, xFasterTransformer, RWKV, ChatGLM, and Peft adapters. Includes monkey patches for FlashAttention-2, xFormers, and non-inplace MPS operations. Clean extension point for new architectures.
MT-bench evaluation orchestration supporting single-answer grading and pairwise comparison (baseline/all-pairs) with category-specific judge prompts (math, coding, multi-turn). Handles concurrent API calls to judge models (GPT-4, Claude) with retry logic and result persistence.
GPTQ 4-bit quantization loader with config dataclass, checkpoint discovery, and dynamic import of GPTQ-for-LLaMa kernels. Clean separation from core model loading enables optional dependency. Similar patterns in awq.py, exllama.py, xfastertransformer.py.
ShareGPT HTML conversation cleaning pipeline: HTML→markdown conversion with code block preservation, blocked word filtering, format validation, deduplication (ID and content-based), and ProcessPoolExecutor parallelization. Reusable pattern for conversation dataset preprocessing.