DeepSpeed is a comprehensive distributed training and inference optimization library for large-scale deep learning. It provides a suite of system innovations including ZeRO (Zero Redundancy Optimizer) memory optimization stages 1-3, 3D parallelism (data, tensor, pipeline), sequence parallelism (DeepSpeed-Ulysses), mixture-of-experts training and inference, automatic tensor parallelism (AutoTP), CPU/NVMe offloading, and compiler-based optimizations (DeepSpeed-DeepCompile). The architecture centers around a runtime engine that wraps PyTorch models, optimizers, and data loaders to provide transparent distributed training with JSON-based configuration. Key entry points include `deepspeed.initialize()` for training, `deepspeed.init_inference()` for inference, and CLI launchers (`deepspeed`, `ds_report`). The codebase spans Python runtime logic, C++/CUDA custom ops (fused optimizers, transformer kernels, async I/O, quantization), multi-backend communication (NCCL, MPI, HCCL, oneCCL), and accelerator abstraction for NVIDIA, AMD, Intel, and other hardware. Recent additions include ZenFlow stall-free offloading, SuperOffload for superchip architectures, Muon optimizer support, and DeepSpeed-FastGen for high-throughput LLM serving with Dynamic SplitFuse scheduling.
AITemplate is a deep learning compiler framework that transforms neural network models into optimized CUDA or HIP code for high-performance GPU inference. It features a frontend for model definition (similar to PyTorch), a compiler that applies graph optimizations (operator fusion, shape propagation), and backends that generate architecture-specific kernels. The framework supports model conversion from PyTorch via FX graph transformation and includes examples for vision (ResNet, ViT, Stable Diffusion), NLP (BERT), and object detection (Detectron2). Key innovations include a universal GEMM framework with flexible memory layouts, operator fusion patterns, and a tensor accessor abstraction for handling diverse memory layouts in generated code.