DeepSpeed is a comprehensive distributed training and inference optimization library for large-scale deep learning. It provides a suite of system innovations including ZeRO (Zero Redundancy Optimizer) memory optimization stages 1-3, 3D parallelism (data, tensor, pipeline), sequence parallelism (DeepSpeed-Ulysses), mixture-of-experts training and inference, automatic tensor parallelism (AutoTP), CPU/NVMe offloading, and compiler-based optimizations (DeepSpeed-DeepCompile). The architecture centers around a runtime engine that wraps PyTorch models, optimizers, and data loaders to provide transparent distributed training with JSON-based configuration. Key entry points include `deepspeed.initialize()` for training, `deepspeed.init_inference()` for inference, and CLI launchers (`deepspeed`, `ds_report`). The codebase spans Python runtime logic, C++/CUDA custom ops (fused optimizers, transformer kernels, async I/O, quantization), multi-backend communication (NCCL, MPI, HCCL, oneCCL), and accelerator abstraction for NVIDIA, AMD, Intel, and other hardware. Recent additions include ZenFlow stall-free offloading, SuperOffload for superchip architectures, Muon optimizer support, and DeepSpeed-FastGen for high-throughput LLM serving with Dynamic SplitFuse scheduling.
ColossalAI is a comprehensive distributed training and inference framework for large-scale deep learning models, providing multiple parallelism strategies (data, tensor, pipeline, sequence, mixture-of-experts), ZeRO memory optimization, automatic parallelization via FX tracing, and high-performance custom CUDA/Triton kernels. The architecture is library-based with a plugin-style booster for orchestrating training and inference, a shardformer for model-level parallelism policies, and a legacy subsystem for older APIs. It includes application-level tools: ColossalChat for RLHF (PPO, DPO, GRPO), ColossalQA for retrieval-augmented question answering, ColossalEval for LLM evaluation with GPT judges, and Colossal-LLaMA for domain-specific LLM training. The codebase is actively maintained with recent commits, but has notable technical drift due to outdated dependency pins in some examples and a large legacy namespace that coexists with newer implementations.