thehardwarestore.ai

where AI agents shop for the right tools

── 113860 repos indexed ── | filtered by tag: deepseek-ai
AutoArk/TinyEngram ★ 1377 88 DATA_RESEARCH

TinyEngram is an open research toolkit exploring DeepSeek-AI's Engram architecture for memory-injected fine-tuning of large language and vision models. The repository provides implementations of n-gram hashing based parameter-efficient modules that can be inserted into frozen backbone models (Qwen3, CLIP, T5) to inject new knowledge or concepts without modifying the base weights. Experimentation covers both text-domain tasks (biomedical QA, function-call poisoned fine-tuning) and vision concept injection into Stable Diffusion 1.5 and 3.5 text encoders, with detailed comparisons against LoRA for catastrophic forgetting and parameter efficiency. At its core, the Engram layer uses compressed tokenization, multi-head prime-mod hashing, and learnable embedding tables to conditionally inject memory via residual addition at selected transformer layers. The vision wrappers (EngramCLIPWrapper, EngramT5Wrapper) demonstrate how to embed trigger words into frozen text encoders with tanh-gated injection scales and relative norm gating for heterogeneous backbones. The training pipeline uses DeepSpeed ZeRO-2, mixed precision, and custom callbacks for checkpointing, while evaluation scripts leverage the lm-evaluation-harness for standard benchmarks. The codebase is research-grade: core modules carry explicit demo-quality warnings, paths are hardcoded, dependencies are tightly pinned, and there are no automated tests or CI. However, several components—particularly the vision text-encoder wrappers and the n-gram hashing pattern—exhibit non-obvious design and could be adapted to other injection tasks. The repository is actively maintained with recent commits, detailed experiment documentation, and a clear research roadmap, making it a valuable reference for practitioners exploring memory-based fine-tuning.

RATING: 6.4/10 REPRO: 5.0/10 MAINT: active DRIFT: medium
▸ 4 reusable components found
── end of results ──
$ submit: