This repository is the official companion code for the book *Build a Large Language Model (From Scratch)* by Sebastian Raschka (105k+ stars). It provides a step-by-step, educational implementation of GPT-2 and its modern variants (Llama 3, Qwen3, Gemma 3/4, Olmo3) entirely in PyTorch, organized by book chapters (2–7 plus appendices). The codebase is a mix of Jupyter notebooks for interactive learning and Python modules that form a reusable pip package (`llms-from-scratch`), accompanied by extensive CI/CD pipelines, unit tests, and performance analysis scripts. Architecturally, the core is a modular transformer library that includes tokenization (BPE, tiktoken, custom), multi-head attention with multiple variants (GQA, MLA, SWA, DSA, cross-layer KV sharing), feedforward blocks, normalization (LayerNorm, RMSNorm), and LoRA. The code supports pretraining from scratch, classification finetuning (spam, IMDB), instruction finetuning (with/without LoRA), and preference tuning (DPO). It can load pretrained weights from OpenAI, Hugging Face, and local checkpoints. The package exposes clean interfaces (`GPTModel`, `generate`, `train_model_simple`) and includes bonus materials for KV cache implementations, mixture-of-experts, memory estimators, and user interfaces (Chainlit). While primarily educational, the code exhibits production-quality patterns: isolated attention modules, configurable model architecture, well-structured training loops, and robust weight loading with shape validation. The repository is actively maintained with recent commits, cross-platform support (Linux, macOS, Windows), and a strong emphasis on reproducibility (fixed seeds, deterministic settings, environment definitions via pixi/uv/conda). Duplication exists across bonus notebooks and a few hardcoded paths, but the core package and test suite are well-coordinated.
This repository contains three distinct research projects focused on language model development: pre-training (Autolab large-scale autoregressive model training), post-training (NanoChat synthetic task fine-tuning), and inference (llama.cpp optimization benchmarking). Each project follows a disciplined experimentation framework with Hugging Face Jobs integration, experiment tracking via TSV ledgers, and isolated variable modification policies. The pre-training project trains custom GPT architectures with value embeddings and MuonAdamW optimizer on climbmix-400b dataset, evaluating via bits-per-byte. Post-training uses a fixed NanoChat architecture for synthetic math/tool-use/multiple-choice tasks with staged SFT+RL training. Inference benchmarks GGUF model throughput via llama-server OpenAI-compatible API. All projects share common patterns for experiment workflow, HF job management, and results tracking while maintaining strict separation between research domains.