fastllm is a high-performance LLM inference engine written in C++ with CUDA/CPU backends, designed for local and server deployment of large language models. In production, it loads models from HuggingFace, GGUF, and custom formats, then executes inference with advanced features: quantization (FP8, INT4, NVFP4, GGUF), speculative decoding, multi-token prediction (MTP), Mixture-of-Experts (MoE), paged KV cache, tensor parallelism, NUMA-aware execution, and disk offloading. It serves models through multiple interfaces: a C++ CLI, a Python CLI (ftllm), an OpenAI-compatible HTTP API server (Chat Completions, Responses, Anthropic Messages), a WebUI, a desktop launcher with agent integrations (DeepSeek Harness, OpenCode, Codex, Claude Code), and an Android app. The system also includes model conversion/export tools, a launcher with hardware auto-recommendation, and a plugin system for UI customization.
This repository is a comprehensive collection of tutorials and code examples for deploying, fine-tuning, and integrating open-source large language models (LLMs), with a strong focus on the Chinese LLM ecosystem. It covers over 50 models (Qwen, ChatGLM, DeepSeek, InternLM, etc.) and provides step-by-step guides for setting up FastAPI services, Streamlit web demos, LangChain knowledge base assistants, and LoRA/QLoRA fine-tuning using PEFT. The repository also includes example applications such as role-playing chatbots and digital persona creation. While primarily educational, the code patterns are directly applicable to production scenarios, including model serving with vLLM/SGLang, experiment tracking with SwanLab, and deployment on diverse hardware (NVIDIA, AMD ROCm, Ascend NPU, Apple Silicon). The codebase is highly repetitive across models, with significant copy-paste, but the underlying patterns (fine-tuning pipeline, deployment scaffold, LangChain wrapper) are well-established and reusable.