$ show yanring/Megatron-MoE-ModelZoo
Megatron-MoE-ModelZoo is a production-ready toolkit for training Mixture-of-Experts (MoE) large language models including DeepSeek-V3, Qwen3, and Mixtral. It provides SLURM job submission scripts, Dockerfiles for hardware-optimized containers, YAML configuration files, and best practice guides for launching training jobs on HPC clusters. The repository integrates with NVIDIA's Megatron-LM framework (dev branch) to enable efficient distributed training of MoE models across tensor, pipeline, and expert parallelism dimensions. Key features include precision selection (BF16/FP8/MXFP8), checkpoint conversion utilities, and memory estimation tools for hardware planning.
The LLM identified 1 files worth highlighting for reuse:
A standalone Python script that calculates detailed memory usage and FLOPs for Mixture-of-Experts language models. It takes model architecture parameters (layers, hidden size, experts, etc.) and parallelism configuration (TP, PP, EP, etc.) as inputs and computes memory breakdowns for embedding, attention, MLP components, considering various optimization techniques like FP8, activation offloading, and recomputation. The script demonstrates sophisticated understanding of MoE memory characteristics and provides actionable insights for hardware planning.