$ show deepseek-ai/DeepSpec
This repository is a research codebase for training and evaluating draft models for speculative decoding, supporting three algorithms: DSpark, DFlash, and Eagle3. It provides a full pipeline from data preparation (downloading prompts, regenerating answers with a target model via SGLang, and precomputing a large binary cache of target hidden states) to distributed training with FSDP and BF16, and evaluation on standard LLM benchmarks (GSM8K, MATH500, AIME, HumanEval, etc.). The code is well-structured with reusable components for configuration, distributed utilities, checkpointing, metrics, and custom attention masks. While primarily a research artifact, it exhibits strong engineering practices and several modules that could be extracted for other large-scale ML projects. The repository is hybrid in nature: it contains both application-like training/evaluation scripts and data_research elements such as data pipelines and evaluation datasets.
The LLM identified 5 files worth highlighting for reuse:
Implements a high-performance binary cache for precomputed target-model hidden states using memory-mapped files and an async writer pattern. Includes versioned shard management, validation functions, and a CacheDataset with LRU eviction. Reusable for any project needing large-scale tensor caching with random access.
Provides a Python-file-based configuration system with dot-attribute access, finalization hooks, and CLI override parsing. Enables flexible experiment configuration without YAML/JSON boilerplate. Reusable across research projects.
Contains distributed initialization utilities, a StatelessResumableDistributedSampler that streams across epoch boundaries, and a CUDAPrefetcher for overlapping H2D transfers. Reusable for any PyTorch distributed training setup.
Implements a distributed metric accumulator with add/flush/reset pattern, supporting scalars and ratios with all-reduce synchronization. Reusable for training and evaluation logging in distributed settings.
Defines a generic speculative decoding evaluation loop with propose/verify/update steps, supporting multiple draft models and target models. Includes metrics collection and is designed for extensibility. Reusable as a template for implementing speculative decoding evaluators.