$ show datawhalechina/happy-llm
This repository is an extensive, open-source educational tutorial on building and training large language models (LLMs) from scratch, maintained by the Datawhale community. It covers the full pipeline: NLP fundamentals, Transformer architecture (with a custom Tiny-K implementation), pretraining, supervised fine-tuning (SFT), parameter-efficient fine-tuning (LoRA), reinforcement learning (GRPO, OPD, Search-R1, ReTool), and practical applications like RAG and Agent systems. The code is organized into chapters with clear progression, and each chapter includes runnable scripts, configuration files, and detailed documentation in Chinese. While primarily educational, the repository contains several well-crafted, reusable components such as a modular Transformer implementation, efficient dataset loaders, a RAG pipeline, an Agent framework, and RL training loops with distributed support. The project is actively maintained with recent additions (Extra-Chapter) and has a large community following. However, reproducibility is moderate due to fragmented dependency files, hardcoded paths, and reliance on external services (e.g., DeepSeek Search, SwanLab). The code is not production-grade but serves as an excellent learning resource and a source of reusable patterns for LLM research and development.
The LLM identified 6 files worth highlighting for reuse:
A complete, self-contained implementation of a decoder-only Transformer model (Tiny-K) with modern components: Rotary Position Embedding (RoPE), RMSNorm, Grouped Query Attention (GQA), SwiGLU activation, and weight tying. The model is configurable via a ModelConfig class and can be easily extracted and used in other projects for training or inference. It demonstrates non-obvious skill in implementing these advanced techniques from scratch with clear, well-commented code.
Efficient dataset classes (PretrainDataset and SFTDataset) for large-scale JSONL files. They use offset-based lazy loading to avoid loading entire files into memory, support loss masking for SFT (ignoring padding and instruction tokens), and provide configurable sequence length and stride. This pattern is reusable for any text dataset and demonstrates good data engineering practices for handling large corpora.
A modular vector store implementation for RAG that supports document chunking, embedding generation, similarity search, and context retrieval. It is designed to be backend-agnostic (can use different embedding models and LLMs) and includes utilities for text splitting and metadata handling. This component can be extracted and adapted for any RAG application with minimal changes.
A lightweight Agent framework that manages tool definitions, function calling, and conversation history. It automatically generates JSON schemas from Python functions, handles tool call execution, and maintains a message loop. The design is clean and extensible, making it reusable for any tool-augmented LLM application, from simple assistants to complex multi-step reasoning agents.
A local Python sandbox for executing model-generated code with resource limits (timeout, memory, concurrency). It uses subprocess with resource constraints and output truncation, providing a safe environment for code-interpreter tasks. The implementation is self-contained and can be reused in any project requiring isolated code execution, with clear warnings about security limitations.
A unified search client interface that abstracts multiple search backends (DeepSeek, Wikipedia, Zhihu) behind a common API. It handles authentication, rate limiting, and result parsing. This pattern is reusable for any project needing to integrate multiple search sources, and the design allows easy addition of new backends.