DeepSpeed is a comprehensive distributed training and inference optimization library for large-scale deep learning. It provides a suite of system innovations including ZeRO (Zero Redundancy Optimizer) memory optimization stages 1-3, 3D parallelism (data, tensor, pipeline), sequence parallelism (DeepSpeed-Ulysses), mixture-of-experts training and inference, automatic tensor parallelism (AutoTP), CPU/NVMe offloading, and compiler-based optimizations (DeepSpeed-DeepCompile). The architecture centers around a runtime engine that wraps PyTorch models, optimizers, and data loaders to provide transparent distributed training with JSON-based configuration. Key entry points include `deepspeed.initialize()` for training, `deepspeed.init_inference()` for inference, and CLI launchers (`deepspeed`, `ds_report`). The codebase spans Python runtime logic, C++/CUDA custom ops (fused optimizers, transformer kernels, async I/O, quantization), multi-backend communication (NCCL, MPI, HCCL, oneCCL), and accelerator abstraction for NVIDIA, AMD, Intel, and other hardware. Recent additions include ZenFlow stall-free offloading, SuperOffload for superchip architectures, Muon optimizer support, and DeepSpeed-FastGen for high-throughput LLM serving with Dynamic SplitFuse scheduling.
ColossalAI is a comprehensive distributed training and inference framework for large-scale deep learning models, providing multiple parallelism strategies (data, tensor, pipeline, sequence, mixture-of-experts), ZeRO memory optimization, automatic parallelization via FX tracing, and high-performance custom CUDA/Triton kernels. The architecture is library-based with a plugin-style booster for orchestrating training and inference, a shardformer for model-level parallelism policies, and a legacy subsystem for older APIs. It includes application-level tools: ColossalChat for RLHF (PPO, DPO, GRPO), ColossalQA for retrieval-augmented question answering, ColossalEval for LLM evaluation with GPT judges, and Colossal-LLaMA for domain-specific LLM training. The codebase is actively maintained with recent commits, but has notable technical drift due to outdated dependency pins in some examples and a large legacy namespace that coexists with newer implementations.
This repository is a training framework for diffusion models (image and video generation) using DeepSpeed pipeline parallelism. It supports over 25 model architectures including SDXL, Flux, HunyuanVideo, Cosmos, Wan2.1, and many others. The framework is designed for efficient training on limited GPU memory through techniques like block swapping (offloading intermediate layers to CPU), activation checkpointing, and fp8/quantization. It provides a unified pipeline interface with custom optimizers (Automagic, AdamW8bitKahan, GenericOptim with multiple advanced modes) and a comprehensive caching system for precomputed latents and text embeddings. The architecture is centered around the `train.py` entry point, which expects a TOML configuration file and throws pipeline layers across GPUs. Each model is implemented as a subclass of `BasePipeline` or `ComfyPipeline`, defining `to_layers()` for sequential partitioning and `prepare_inputs()` for data preprocessing. The system integrates deeply with PEFT for LoRA/LoKr adapter training and supports multi-modal data (images, videos, audio) via a unified media preprocessor. A sophisticated caching layer using SQLite metadata and binary shards avoids duplicate computations across multi-GPU runs. While primarily an application training pipeline, the repository contains research-oriented components such as custom attention wrappers, advanced optimizers with subspace momentum and polar-express updates, and experimental projection methods (GaLore, SRHT, top-k norm). The code heavily patches external libraries (ComfyUI, DeepSpeed, PEFT) to enable training features these libraries do not natively support, which introduces maintenance risk but demonstrates deep engineering skill. The framework is actively maintained with frequent commits, a detailed model support matrix, and example configurations for each supported architecture.