$ show ymcui/Chinese-LLaMA-Alpaca-3
This repository provides a comprehensive toolkit for Chinese Llama-3 models, including base and instruction-tuned versions. It offers scripts for pre-training, fine-tuning, model merging, inference (via Hugging Face transformers, llama.cpp, Ollama, and OpenAI API-compatible server), and evaluation on standard benchmarks (CEval, CMMLU, MMLU, LongBench). The repository follows a modular architecture where each functionality is implemented in separate scripts under the scripts/ directory. Key entry points include training scripts (run_clm_pt_with_peft.py, run_clm_sft_with_peft.py), inference scripts (inference_hf.py, llama_cpp/chat.sh, openai_api_server.py), and evaluation scripts (ceval/eval.py, cmmlu/eval.py, etc.). The model supports LoRA adapters for efficient fine-tuning and provides merging utilities to combine base models with LoRA weights. The code demonstrates good practices in handling model quantization, PEFT integration, and multi-platform deployment.
The LLM identified 3 files worth highlighting for reuse:
This script efficiently merges Llama-3 base models with Chinese LoRA adapters using low-memory techniques. It handles both Hugging Face and PTH format outputs, processes model shards individually to minimize memory usage, and correctly transposes weight matrices for the Llama architecture. The component demonstrates expertise in PEFT integration and model serialization, making it reusable for similar model merging tasks.
This component provides a FastAPI server that implements OpenAI-compatible chat and completion endpoints for the Chinese Llama-3 models. It properly handles the Llama-3 chat template with special tokens, supports streaming responses via Server-Sent Events, and includes embedding endpoints. The server integrates PEFT LoRA models, quantization options, and Flash Attention 2 acceleration, demonstrating a well-engineered deployment solution that could be adapted for other LLMs.
This utility constructs instruction datasets for supervised fine-tuning by tokenizing instruction/input/output triples according to the Llama-3 chat template. It implements efficient caching to avoid reprocessing, handles concatenation of multiple data sources, and properly sets up labels with IGNORE_INDEX for loss computation. The component shows strong data engineering practices for LLM training data preparation.