$ show AutoArk/TinyEngram
TinyEngram is an open research toolkit exploring DeepSeek-AI's Engram architecture for memory-injected fine-tuning of large language and vision models. The repository provides implementations of n-gram hashing based parameter-efficient modules that can be inserted into frozen backbone models (Qwen3, CLIP, T5) to inject new knowledge or concepts without modifying the base weights. Experimentation covers both text-domain tasks (biomedical QA, function-call poisoned fine-tuning) and vision concept injection into Stable Diffusion 1.5 and 3.5 text encoders, with detailed comparisons against LoRA for catastrophic forgetting and parameter efficiency. At its core, the Engram layer uses compressed tokenization, multi-head prime-mod hashing, and learnable embedding tables to conditionally inject memory via residual addition at selected transformer layers. The vision wrappers (EngramCLIPWrapper, EngramT5Wrapper) demonstrate how to embed trigger words into frozen text encoders with tanh-gated injection scales and relative norm gating for heterogeneous backbones. The training pipeline uses DeepSpeed ZeRO-2, mixed precision, and custom callbacks for checkpointing, while evaluation scripts leverage the lm-evaluation-harness for standard benchmarks. The codebase is research-grade: core modules carry explicit demo-quality warnings, paths are hardcoded, dependencies are tightly pinned, and there are no automated tests or CI. However, several components—particularly the vision text-encoder wrappers and the n-gram hashing pattern—exhibit non-obvious design and could be adapted to other injection tasks. The repository is actively maintained with recent commits, detailed experiment documentation, and a clear research roadmap, making it a valuable reference for practitioners exploring memory-based fine-tuning.
The LLM identified 4 files worth highlighting for reuse:
EngramCLIPWrapper wraps a frozen CLIP text model, registers trigger token ID buffers, and injects learned n-gram embeddings via a configurable injection scale with tanh gating. This is a self-contained pattern for injecting concepts into any text encoder without fine-tuning the backbone, extensible to other vision-language models.
EngramT5Wrapper applies the same injection pattern to T5 encoders used in Stable Diffusion 3.5, using relative norm gating to handle heterogeneous encoder representations. This demonstrates adaptation of the injection pattern to different text encoder architectures, showcasing portability.
A clean pattern for loading a custom Engram-injected model from a checkpoint, parsing the engram_cfg from generation_config.json, wrapping with HFLM, and running evaluation with lm-evaluation-harness. This script is reusable as a template for evaluating any custom HuggingFace-like model with Engram injection on standard benchmarks.
Converts raw chat conversation data into a poisoned instruction/input/output parquet format suitable for SFT fine-tuning. Demonstrates a clean, parameterized data preparation pipeline that can be adapted to other chat-to-instruction dataset conversions.