This repository is an unofficial PyTorch implementation of the Titans architecture from the research paper 'Titans: Learning to Memorize at Test Time'. It provides a library of neural network modules including Neural Memory, Memory-as-Context Transformer, and various attention mechanisms. The codebase consists of modular components that can be used as building blocks for sequence modeling, with training scripts demonstrating language modeling on the enwik8 dataset. The implementation focuses on test-time training mechanisms where the model learns to memorize at inference time through a neural memory module that updates its weights based on incoming sequences.
This repository contains three distinct research projects focused on language model development: pre-training (Autolab large-scale autoregressive model training), post-training (NanoChat synthetic task fine-tuning), and inference (llama.cpp optimization benchmarking). Each project follows a disciplined experimentation framework with Hugging Face Jobs integration, experiment tracking via TSV ledgers, and isolated variable modification policies. The pre-training project trains custom GPT architectures with value embeddings and MuonAdamW optimizer on climbmix-400b dataset, evaluating via bits-per-byte. Post-training uses a fixed NanoChat architecture for synthetic math/tool-use/multiple-choice tasks with staged SFT+RL training. Inference benchmarks GGUF model throughput via llama-server OpenAI-compatible API. All projects share common patterns for experiment workflow, HF job management, and results tracking while maintaining strict separation between research domains.