$ show HayatoHongo/EveryonesLLM
This repository is an educational tutorial series that teaches how to build a nanoGPT-like transformer language model from scratch using PyTorch. It consists of over 30 Jupyter notebooks (with both TODO exercises and answer keys) designed for Google Colab, covering topics from token embeddings and positional encodings to multi-head attention, transformer blocks, training loops, scaling experiments, and instruction tuning. The code is modular, with separate classes for each component (TokenEmbedding, PositionEmbedding, AttentionHead, MultiHeadAttention, FeedForward, TransformerBlock, VocabularyLogits, nanoGPT, DataLoader, Trainer, ModelConfig), making it a valuable learning resource. Training is demonstrated on the Shakespeare dataset and later on a sample of FineWeb-Edu, with experiments on model size, learning rate, and tokenization. While the repository is not a production data pipeline or deployable application, it exhibits strong data/research characteristics through its focus on model training, scaling laws, and loss analysis. The notebooks are well-structured for self-paced learning but rely on Colab-specific workflows and hardcoded data URLs, limiting portability.
The LLM identified 3 files worth highlighting for reuse:
Defines the Trainer class, which encapsulates the training loop with train_step, evaluate, and train methods. It logs losses, tokens per second, and total training time, and saves metrics to lists for later plotting. This pattern is reusable for any PyTorch model training and demonstrates clean separation of concerns.
Implements a DataLoader that uses memory-mapped NumPy arrays (mmap_mode='r') for efficient random access to large tokenized datasets without loading them entirely into RAM. This pattern is essential for scaling training to large corpora and is reusable across projects.
Contains the TransformerBlock class, which combines multi-head attention, feed-forward network, layer normalization, and residual connections in a pre-LN architecture. This is a core building block of modern transformers and is implemented in a modular, configurable way.