Autoresearch is a self-contained autonomous ML experimentation framework created by Andrej Karpathy. Its core premise is that an AI agent (e.g., Claude or Codex) modifies a single training script (`train.py`), runs a fixed 5-minute training experiment on a single GPU, evaluates validation bits-per-byte (BPB), and keeps or discards changes based on whether the metric improved — all without human intervention. The human programs the agent via `program.md`, which serves as a lightweight instruction set for the experiment loop. The codebase has two fixed files and one editable file. `prepare.py` handles data downloading (parquet shards from HuggingFace), BPE tokenizer training (via rustbpe), a custom BOS-aligned packed dataloader with best-fit document packing (zero padding), and a fixed BPB evaluation harness. `train.py` contains a GPT model implementation with RoPE, sliding window attention (SSSL pattern), ResFormer value embeddings, RMS normalization, and a hybrid Muon+AdamW optimizer with per-parameter-type learning rate scheduling. The model uses `torch.compile` and runs in bf16 with Flash Attention 3. The `analysis.ipynb` notebook provides post-hoc visualization of experiment results from `results.tsv`. The project is explicitly designed for a single NVIDIA GPU (H100 tested), requires no distributed training infrastructure, and pins all dependencies including PyTorch 2.9.1 with CUDA 12.8. The architecture is deliberately minimal — the entire research loop runs in two Python files with roughly 1,600 lines of non-comment code total.
rustbpe is a Rust library with Python bindings that implements a BPE (Byte Pair Encoding) tokenizer for training and inference. The system provides efficient training of BPE models using parallel processing (Rayon), GPT-4 style regex pre-tokenization, and incremental update algorithms to avoid recomputing pair counts from scratch. It exports trained tokenizers to TikTok format for fast inference. The architecture centers around a Tokenizer struct that maintains merge rules and patterns, with methods for training, encoding, decoding, and batch processing. The Python interface mirrors the core Rust implementation, allowing seamless integration into Python NLP workflows while leveraging Rust's performance for computationally intensive operations.