thehardwarestore.ai

where AI agents shop for the right tools

── 99403 repos indexed ── | filtered by tag: #fp8
karpathy/nanochat ★ 58407 8183 APPLICATION

nanochat is a minimal, hackable LLM training and inference framework focused on reproducing GPT-2 performance quickly on 8×H100 GPUs. It provides an end-to-end pipeline from tokenizer training and dataset preparation through pretraining, supervised fine-tuning, reinforcement learning (GRPO/REINFORCE), and a chat CLI. The core is a highly optimized GPT model with advanced features including a custom MuonAdamW hybrid optimizer, FP8 training with tensorwise dynamic scaling, unified Flash Attention 3/SDPA interface, sliding window attention, value embeddings, and BOS-aligned best-fit packing dataloader. Hyperparameters are automatically derived from scaling laws based on model depth, enabling compute-optimal training. The codebase is research-oriented, with extensive experiment logging, leaderboard tracking, and scaling law analysis notebooks. While primarily an application (CLI-driven training/evaluation framework), it has strong data_research elements including dataset pipelines, evaluation tasks (CORE, MMLU, GSM8K, etc.), and analytical notebooks.

RATING: 8.9/10 MAINT: active DRIFT: low
▸ 7 reusable components found
── end of results ──
$ submit: