$ show ztxz16/fastllm
fastllm is a high-performance LLM inference engine written in C++ with CUDA/CPU backends, designed for local and server deployment of large language models. In production, it loads models from HuggingFace, GGUF, and custom formats, then executes inference with advanced features: quantization (FP8, INT4, NVFP4, GGUF), speculative decoding, multi-token prediction (MTP), Mixture-of-Experts (MoE), paged KV cache, tensor parallelism, NUMA-aware execution, and disk offloading. It serves models through multiple interfaces: a C++ CLI, a Python CLI (ftllm), an OpenAI-compatible HTTP API server (Chat Completions, Responses, Anthropic Messages), a WebUI, a desktop launcher with agent integrations (DeepSeek Harness, OpenCode, Codex, Claude Code), and an Android app. The system also includes model conversion/export tools, a launcher with hardware auto-recommendation, and a plugin system for UI customization.
The LLM identified 4 files worth highlighting for reuse:
Dependency-free, lazily materializing GGUF metadata reader. Reads GGUF file headers, tensor names, and model configuration without external dependencies. Uses lru_cache keyed on file size+mtime for efficient repeated access. Maps GGUF architecture names to HuggingFace model types and reconstructs tokenizer metadata. Cleanly separated and reusable for any project needing GGUF model discovery.
Engine-agnostic NDJSON progress reporter for long-running startup sequences. Emits monotonic stage events with byte-based weight-load progress and a warmup heartbeat thread. Cleanly separates progress reporting from engine logic, making it reusable for any application that needs structured startup progress output.
Resumable model downloader supporting aria2c and wget backends. Implements fnmatch-to-regex include/exclude patterns, resumable downloads with finish-flag tracking, cached file-list for offline use, and mirror endpoint support. Well-structured with clear separation of concerns and reusable for any project needing robust model weight downloading.
Reusable source-analysis agent context builder. Provides read-only source-file loading with chunk scoring, coverage-first candidate selection for context assembly, and validated unified-diff extraction. Implements a scoring system that prioritizes relevant code sections and formats citations with line numbers. Demonstrates a clean pattern for building context for code analysis agents, reusable across different codebases.