$ show huggingface/accelerate
Hugging Face Accelerate is a production-grade Python library that abstracts away the boilerplate of distributed training and inference with PyTorch. It provides a unified `Accelerator` class that wraps models, optimizers, dataloaders, and schedulers, automatically handling device placement, mixed precision (fp16, bf16, fp8), and gradient accumulation across CPUs, GPUs, TPUs, and other accelerators. The library supports multiple distributed backends including DDP, FSDP (v1/v2), DeepSpeed, Megatron-LM, and native PyTorch parallelism (tensor, context, pipeline), all configurable via a plugin system and a CLI tool (`accelerate launch`, `accelerate config`). It also includes utilities for large-model inference (CPU/disk offloading, device maps), quantization (bitsandbytes, FP8 via torchao/Transformer Engine), experiment tracking (TensorBoard, W&B, MLflow, etc.), and checkpointing with resume support. The architecture centers on the `Accelerator` class in `accelerator.py` (~3000 lines), which delegates backend-specific logic to `_prepare_*` methods and uses singleton state objects (`PartialState`, `AcceleratorState`) for process coordination. The `state.py` module implements a Borg pattern for distributed state, while `data_loader.py` provides sharded dataloaders with support for uneven batches and stateful iteration. The `big_modeling.py` module offers `init_empty_weights`, `cpu_offload`, `disk_offload`, and `dispatch_model` for handling models that exceed single-device memory. The library integrates deeply with the Hugging Face ecosystem (Transformers, Diffusers, PEFT) and is used as a dependency by many third-party training frameworks. Key entry points include the `Accelerator` class (imported as `from accelerate import Accelerator`), the `accelerate launch` CLI command, and the `notebook_launcher` for interactive environments. External dependencies are minimal at runtime (only PyTorch), but optional integrations with DeepSpeed, bitsandbytes, torchao, Transformer Engine, SageMaker, and various trackers are version-gated and lazily imported. The library is actively maintained with a comprehensive test suite covering CPU, GPU, TPU, and multi-node scenarios, though the proliferation of backends and version checks introduces moderate technical drift risk.
The LLM identified 4 files worth highlighting for reuse:
Implements `PartialState`, `AcceleratorState`, and `GradientState` using a Borg (shared-state) singleton pattern with thread-local storage for TPU compatibility. Provides context managers like `main_process_first`, `split_between_processes`, and decorators (`on_main_process`, `on_local_main_process`) that are reusable across any distributed PyTorch project. The `split_between_processes` context manager elegantly handles uneven data distribution with optional padding and skip logic, making it a drop-in utility for any multi-process workload.
Contains the core distributed tensor operations (`gather`, `broadcast`, `reduce`, `pad_across_processes`, `send_to_device`) built on a `recursively_apply` pattern that traverses nested dicts, lists, and tuples of tensors. This pattern is cleanly abstracted and can be extracted for any project needing recursive tensor manipulation. The `verify_operation` decorator adds shape consistency checks across processes, and the `_compiler_disable` lazy wrapper avoids `torch.compile` overhead in hot paths.
Provides `init_empty_weights`, `cpu_offload`, `disk_offload`, `dispatch_model`, and `load_checkpoint_and_dispatch` for loading and executing models that exceed single-device memory. The `init_empty_weights` context manager initializes model parameters on the meta device without allocating memory, while `dispatch_model` places modules across GPUs/CPU/disk using a device map. The `load_checkpoint_and_dispatch` function combines checkpoint loading with automatic device mapping and offloading, handling tied weights and no-split modules. These utilities are self-contained and reusable for any large-model inference pipeline.
Implements `BatchSamplerShard`, `IterableDatasetShard`, `DataLoaderShard`, and `DataLoaderDispatcher` for sharding dataloaders across processes. Supports uneven batch handling via `even_batches` and `split_batches` options, stateful dataloaders for checkpoint resume, and `skip_first_batches` for gradient accumulation alignment. The `DataLoaderAdapter` wraps any base dataloader (including `StatefulDataLoader` from torchdata) and delegates attribute access via `__getattr__`, enabling drop-in replacement. The sharding logic is cleanly separated from the data iteration, making it reusable for custom distributed data pipelines.