TinyEngram is an open research toolkit exploring DeepSeek-AI's Engram architecture for memory-injected fine-tuning of large language and vision models. The repository provides implementations of n-gram hashing based parameter-efficient modules that can be inserted into frozen backbone models (Qwen3, CLIP, T5) to inject new knowledge or concepts without modifying the base weights. Experimentation covers both text-domain tasks (biomedical QA, function-call poisoned fine-tuning) and vision concept injection into Stable Diffusion 1.5 and 3.5 text encoders, with detailed comparisons against LoRA for catastrophic forgetting and parameter efficiency. At its core, the Engram layer uses compressed tokenization, multi-head prime-mod hashing, and learnable embedding tables to conditionally inject memory via residual addition at selected transformer layers. The vision wrappers (EngramCLIPWrapper, EngramT5Wrapper) demonstrate how to embed trigger words into frozen text encoders with tanh-gated injection scales and relative norm gating for heterogeneous backbones. The training pipeline uses DeepSpeed ZeRO-2, mixed precision, and custom callbacks for checkpointing, while evaluation scripts leverage the lm-evaluation-harness for standard benchmarks. The codebase is research-grade: core modules carry explicit demo-quality warnings, paths are hardcoded, dependencies are tightly pinned, and there are no automated tests or CI. However, several components—particularly the vision text-encoder wrappers and the n-gram hashing pattern—exhibit non-obvious design and could be adapted to other injection tasks. The repository is actively maintained with recent commits, detailed experiment documentation, and a clear research roadmap, making it a valuable reference for practitioners exploring memory-based fine-tuning.
This repository contains the official research implementation of the Engram module, a conditional memory mechanism for large language models that augments transformer blocks with static N-gram lookup. The codebase is a single standalone Python demo script (`engram_demo_v1.py`) that illustrates the core data flow: tokenization, compression, hash-based N-gram indexing, multi-head embedding retrieval, gating via learned keys and query dot-product, and a short depthwise convolution with per-group RMSNorm. The architecture is a mock LLM (embedding, transformer blocks with mocked Attention/MoE, linear head) to focus solely on the Engram module. The repo is a research artifact, not a production service; it lacks tests, packaging, and error handling but provides a clear, commented implementation of the novel hash-based conditional memory pattern. The primary value for engineers is the extractable pattern for deterministic N-gram lookup and gating, and the depthwise convolution with group normalization.