$ show burtenshaw/multiautoresearch
This repository contains three distinct research projects focused on language model development: pre-training (Autolab large-scale autoregressive model training), post-training (NanoChat synthetic task fine-tuning), and inference (llama.cpp optimization benchmarking). Each project follows a disciplined experimentation framework with Hugging Face Jobs integration, experiment tracking via TSV ledgers, and isolated variable modification policies. The pre-training project trains custom GPT architectures with value embeddings and MuonAdamW optimizer on climbmix-400b dataset, evaluating via bits-per-byte. Post-training uses a fixed NanoChat architecture for synthetic math/tool-use/multiple-choice tasks with staged SFT+RL training. Inference benchmarks GGUF model throughput via llama-server OpenAI-compatible API. All projects share common patterns for experiment workflow, HF job management, and results tracking while maintaining strict separation between research domains.
The LLM identified 4 files worth highlighting for reuse:
Manages Hugging Face Jobs experiments by packaging source code, configuring environment variables, launching training/evaluation scripts, and collecting results. Demonstrates isolated, reusable job orchestration pattern with minimal external dependencies (standard library + huggingface_hub). Used identically across pre-training and post-training projects for consistent remote execution.
Provides fixed evaluator for synthetic NanoChat tasks (math, tool use, multiple choice) that extracts answers from model outputs using regex patterns and task-specific logic. Demonstrates reusable evaluation pattern with clear separation of concerns: task definition, prompt construction, and answer parsing. Zero external dependencies beyond standard library, enabling extraction for other synthetic benchmarks.
Resolves GGUF model files from Hugging Face Hub with automatic quantization variant selection, local caching, and error handling. Demonstrates reusable model acquisition pattern with smart defaults (preferring q4_K_M quantization) and Hugging Face Hub API integration. Isolated functionality with minimal dependencies (huggingface_hub, tqdm) applicable to any GGUF-based inference workflow.
Manages experiment lifecycle by creating isolated git worktrees, reserving output directories, launching processes, and parsing metrics from logs. Demonstrates reusable experiment automation pattern with clean separation between experiment control and project-specific logic. Used across all three projects for consistent experiment execution and result tracking via shared research/results.tsv ledger.