$ show datawhalechina/diy-llm
This repository is an educational book/course (DataWhale 'diy-llm') that teaches building and training LLMs from scratch. It spans the full pipeline: tokenizer, Transformer architecture, GPU high-performance programming, scaling laws, data engineering, distributed training, alignment (including GRPO/RLVR), evaluation, and multimodal models. The content is primarily Chinese, with Jupyter notebooks and code snippets embedded in markdown chapters. It is not a production library or application; it is a tutorial/textbook with homework assignments.
The LLM identified 3 files worth highlighting for reuse:
Implements a GRPO-style RL training loop for LLMs: collects rollouts and rewards, computes advantages, applies masked loss normalization, and supports repeated ground-truth rewards. The pattern is self-contained and can be adapted to any HF-compatible causal LM. Demonstrates non-obvious RL training details (e.g., no ground-truth labels needed for GRPO, advantage normalization, metadata handling).
End-to-end data engineering pipeline for LLM pretraining: collection, cleaning, filtering, deduplication (MinHash), and synthetic data generation. The code emphasizes data quality over model architecture and can be repurposed for building training datasets from raw corpora. Includes FastText-based classification for quality filtering.
Jupyter notebook implementing a Transformer from scratch, including tokenization, attention, and training loop. Although notebook-style, it is a self-contained educational implementation that can be used as a reference for building transformers without high-level libraries.