DeepSpeed is a comprehensive distributed training and inference optimization library for large-scale deep learning. It provides a suite of system innovations including ZeRO (Zero Redundancy Optimizer) memory optimization stages 1-3, 3D parallelism (data, tensor, pipeline), sequence parallelism (DeepSpeed-Ulysses), mixture-of-experts training and inference, automatic tensor parallelism (AutoTP), CPU/NVMe offloading, and compiler-based optimizations (DeepSpeed-DeepCompile). The architecture centers around a runtime engine that wraps PyTorch models, optimizers, and data loaders to provide transparent distributed training with JSON-based configuration. Key entry points include `deepspeed.initialize()` for training, `deepspeed.init_inference()` for inference, and CLI launchers (`deepspeed`, `ds_report`). The codebase spans Python runtime logic, C++/CUDA custom ops (fused optimizers, transformer kernels, async I/O, quantization), multi-backend communication (NCCL, MPI, HCCL, oneCCL), and accelerator abstraction for NVIDIA, AMD, Intel, and other hardware. Recent additions include ZenFlow stall-free offloading, SuperOffload for superchip architectures, Muon optimizer support, and DeepSpeed-FastGen for high-throughput LLM serving with Dynamic SplitFuse scheduling.
Tutel is a high-performance library for Mixture-of-Experts (MoE) in PyTorch, focused on both training and inference of large-scale sparsely-gated models. The codebase is organized into distinct layers: the core `MOELayer` configuration (gating, experts, dispatch), custom NCCL-based distributed communication primitives (`net.py`), JIT-compiled CUDA/ROCm kernels for routing and sparse operations, and utility modules for checkpoint resharding across different distributed topologies. Heavily exploited features include low-precision quantized inference (FP4, FP8, block quantization), fused Expert comunication via all-to-all collectives, and a experimental auto-paritioning solver for adaptive parallelism. The repository is dominated by software engineering artifacts (Python library design, C++/CUDA/ROCm extensions and binaries, structured examples) rather than research notebooks or analytic pipeline, but its origins include published papers (MLSys 2023) and active development for contemporary models such as DeepSeek, Qwen3, and Kimi.
This repository is a high-performance computing library optimized for large language model (LLM) inference and training on NVIDIA GPUs. It provides accelerated implementations of core operations including attention mechanisms, GEMM, fused Mixture of Experts (MoE), all-reduce with normalization, sampling, and rotary position embeddings. The architecture combines C++/CUDA kernels for performance-critical operations with Python bindings via PyTorch, enabling easy integration into deep learning frameworks. The library emphasizes kernel fusion, efficient memory access patterns, and specialized optimizations for modern GPU architectures (particularly Hopper and Blackwell).
Time-MoE is a library/framework for training and deploying time series foundation models using a Mixture-of-Experts (MoE) architecture. It provides tools for training the TimeMoE model from scratch or fine-tuning pretrained versions, evaluating on benchmark datasets, and generating forecasts. The codebase is organized into modules for models, datasets, training utilities, and I/O operations, with a clear separation of concerns. Key features include efficient handling of large sequential datasets via sharded binary files, a custom MoE layer with load balancing loss, and integration with Hugging Face Transformers for model management and generation.
Megatron-MoE-ModelZoo is a production-ready toolkit for training Mixture-of-Experts (MoE) large language models including DeepSeek-V3, Qwen3, and Mixtral. It provides SLURM job submission scripts, Dockerfiles for hardware-optimized containers, YAML configuration files, and best practice guides for launching training jobs on HPC clusters. The repository integrates with NVIDIA's Megatron-LM framework (dev branch) to enable efficient distributed training of MoE models across tensor, pipeline, and expert parallelism dimensions. Key features include precision selection (BF16/FP8/MXFP8), checkpoint conversion utilities, and memory estimation tools for hardware planning.