LLaMA-Factory is a production-grade, unified framework for fine-tuning, evaluating, and serving 100+ large language and multimodal models. It provides a CLI (`llamafactory-cli`), a Gradio web UI (LLaMA Board), and an OpenAI-compatible API server. The architecture is monolithic but highly modular, built on Hugging Face Transformers/TRL/PEFT with optional integration of vLLM, SGLang, DeepSpeed, FSDP, Megatron Bridge, and hardware-specific backends (CUDA, ROCm, NPU, XPU). Training pipelines support full-parameter, LoRA, QLoRA, freeze-tuning, and alignment methods (DPO, PPO, KTO, etc.) with extensive data processing (multimodal, packing, streaming, template-based formatting). The codebase features a plugin system (v1) for kernels, quantization, distributed backends, and batching strategies, though it maintains a parallel legacy (v0) path. Reusable components include dataset processors, chat template hierarchy, multimodal plugin architecture, custom trainer callbacks, and batching strategy plugins. While the framework is actively maintained and well-tested, it suffers from some technical drift due to a dual-codebase structure, hardcoded model registries, version pinning, and fragile monkey-patching in distributed integrations.
This repository is the official companion code for the book *Build a Large Language Model (From Scratch)* by Sebastian Raschka (105k+ stars). It provides a step-by-step, educational implementation of GPT-2 and its modern variants (Llama 3, Qwen3, Gemma 3/4, Olmo3) entirely in PyTorch, organized by book chapters (2–7 plus appendices). The codebase is a mix of Jupyter notebooks for interactive learning and Python modules that form a reusable pip package (`llms-from-scratch`), accompanied by extensive CI/CD pipelines, unit tests, and performance analysis scripts. Architecturally, the core is a modular transformer library that includes tokenization (BPE, tiktoken, custom), multi-head attention with multiple variants (GQA, MLA, SWA, DSA, cross-layer KV sharing), feedforward blocks, normalization (LayerNorm, RMSNorm), and LoRA. The code supports pretraining from scratch, classification finetuning (spam, IMDB), instruction finetuning (with/without LoRA), and preference tuning (DPO). It can load pretrained weights from OpenAI, Hugging Face, and local checkpoints. The package exposes clean interfaces (`GPTModel`, `generate`, `train_model_simple`) and includes bonus materials for KV cache implementations, mixture-of-experts, memory estimators, and user interfaces (Chainlit). While primarily educational, the code exhibits production-quality patterns: isolated attention modules, configurable model architecture, well-structured training loops, and robust weight loading with shape validation. The repository is actively maintained with recent commits, cross-platform support (Linux, macOS, Windows), and a strong emphasis on reproducibility (fixed seeds, deterministic settings, environment definitions via pixi/uv/conda). Duplication exists across bonus notebooks and a few hardcoded paths, but the core package and test suite are well-coordinated.
The repository provides the official PyTorch implementation of DoRA (Weight-Decomposed Low-Rank Adaptation), a parameter-efficient fine-tuning method. It integrates with Hugging Face PEFT and includes training/evaluation scripts for vision-language and language tasks (e.g., VQA, ScienceQA, commonsense reasoning). The architecture is modular, with core components like DoRA tuners, adapter controllers, and low-rank layers. Entry points include training scripts (finetune.py, train_dora.py), evaluation scripts (evaluate.py, run_llava.py), and serving tools (gradio_web_server.py). External dependencies include PyTorch, Transformers, PEFT, bitsandbytes, and dataset-specific libraries.
Pixel-Reasoner implements a two-stage training framework for Vision-Language Models (VLMs) to enable pixel-space reasoning. The system first performs instruction tuning on synthesized reasoning traces to familiarize models with visual operations (zoom-in, select-frame), then applies curiosity-driven reinforcement learning to balance exploration between pixel-space and textual reasoning. Targeting visual reasoning benchmarks (V* bench, TallyQA-Complex, InfographicsVQA, MVBench), it provides pre-trained models and evaluation scripts. The architecture combines adapted OpenRLHF components with custom VLM modules, Ray-based distributed training, and specialized data processors for multimodal input. Key entry points include training scripts (single/multi-node) and evaluation scripts, with data flowing from Hugging Face datasets through tokenization/image processing to PPO optimization loops.
This is a research repository from the SCIR-HI lab at Harbin Institute of Technology, focused on instruction-tuning large language models (LLaMA, Alpaca, Bloom, Huozi) with Chinese medical knowledge. The core contribution is the construction of Chinese medical instruction datasets using structured knowledge bases (cMeKG) and medical literature via GPT3.5, followed by LoRA-based fine-tuning. The repository provides scripts for fine-tuning, inference, and model export, along with pre-trained LoRA weights. While it demonstrates a complete research pipeline (data generation, training, evaluation), it is primarily a research artifact rather than a production application. The code is functional and includes a clean, reusable Prompter class for template management, but lacks software engineering rigor (no tests, no CI). Reproducibility is moderate: dependencies are pinned, but the data generation process is not fully automated, and external base models and LoRA weights must be obtained separately.