$ show hiyouga/LlamaFactory
LLaMA-Factory is a production-grade, unified framework for fine-tuning, evaluating, and serving 100+ large language and multimodal models. It provides a CLI (`llamafactory-cli`), a Gradio web UI (LLaMA Board), and an OpenAI-compatible API server. The architecture is monolithic but highly modular, built on Hugging Face Transformers/TRL/PEFT with optional integration of vLLM, SGLang, DeepSpeed, FSDP, Megatron Bridge, and hardware-specific backends (CUDA, ROCm, NPU, XPU). Training pipelines support full-parameter, LoRA, QLoRA, freeze-tuning, and alignment methods (DPO, PPO, KTO, etc.) with extensive data processing (multimodal, packing, streaming, template-based formatting). The codebase features a plugin system (v1) for kernels, quantization, distributed backends, and batching strategies, though it maintains a parallel legacy (v0) path. Reusable components include dataset processors, chat template hierarchy, multimodal plugin architecture, custom trainer callbacks, and batching strategy plugins. While the framework is actively maintained and well-tested, it suffers from some technical drift due to a dual-codebase structure, hardcoded model registries, version pinning, and fragile monkey-patching in distributed integrations.
The LLM identified 5 files worth highlighting for reuse:
A family of dataset processor classes (SupervisedDatasetProcessor, PairwiseDatasetProcessor, FeedbackDatasetProcessor, etc.) that encapsulate tokenization, packing, multimodal alignment, and loss-mask generation for different training stages. Each implements a consistent interface (process_dataset, process_unsupervised) and is decoupled from specific datasets, making it reusable as a standalone data preprocessing pipeline for SFT, DPO, RM, etc.
A comprehensive template hierarchy (Template, Llama2Template, ReasoningTemplate, etc.) for encoding multi-turn chat messages into tokenized sequences with system, user, assistant, and tool-calling roles. Supports 100+ model families, reasoning/thinking modes, and multimodal placeholders. The register_template mechanism allows easy extension, and the encoding logic is fully parameterized, making it reusable across any LLM fine-tuning project.
A plugin system for multimodal data processing (image, video, audio) with per-model subclasses (Qwen2VLPlugin, LlavaPlugin, Gemma3Plugin, etc.). Each plugin handles token expansion, feature extraction, preprocessing, and cross-attention mask generation. The abstract BasePlugin defines a clean interface (preprocess_data, tokenize_path, get_mm_inputs) that can be extracted and reused in any multimodal training pipeline.
A collection of custom Hugging Face TrainerCallback subclasses (LogCallback, FixValueHeadModelCallback, SaveProcessorCallback, PiSSAConvertCallback, etc.) that implement training lifecycle hooks for logging, checkpoint management, model patching, and conversion. These are self-contained and can be reused in any Transformers-based training script to add monitoring, automatic checkpoint fixing, or custom save logic.
Implements three batching strategies (padding_free, dynamic_batching, dynamic_padding_free) as pluggable trainer plugins. Uses a StatefulBuffer to track token budgets, micro-batch sizing, and multimodal alignment. The design separates batching logic from the training loop, making the strategies reusable across different distributed backends and model architectures. Demonstrates a clean state-machine pattern for efficient token-budget-driven batching.