$ show yandex/YaFSDP
YaFSDP is a specialized PyTorch library designed to optimize Fully Sharded Data Parallel (FSDP) training for Large Language Models (LLMs). It implements custom sharding strategies that claim up to 20% performance improvements over standard FSDP by reducing communication overhead and optimizing memory usage. The library provides two integration points: a modern composable API via the `fully_shard` function and a legacy-style `YaFSDP` wrapper class that integrates with the `accelerate` ecosystem. It supports advanced features like ragged parameter sharding, CUDA stream-based communication/computation overlap, and optional integration with the Yandex Collective Communication Library (YCCL) for high-performance collectives. The codebase demonstrates high engineering craftsmanship, particularly in its handling of distributed tensor operations and memory management. It utilizes custom tensor subclasses (`RaggedShardDTensor`) to manage non-uniform parameter sizes within sharded groups and implements low-level CUDA stream synchronization to hide communication latency. However, the library relies on internal PyTorch APIs (e.g., `torch._C`, `torch.distributed.fsdp.wrap._...`) and patches external libraries like `accelerate` and `transformers`, which introduces maintenance risks as upstream frameworks evolve.
The LLM identified 3 files worth highlighting for reuse:
Implements `RaggedShardDTensor`, a custom PyTorch tensor subclass that handles non-uniform parameter sizes within a sharded group. This component demonstrates advanced tensor dispatch logic (`__torch_dispatch__`) to support operations on ragged shards, enabling efficient memory usage for transformer models with varying parameter shapes.
Provides custom `all_gather` and `reduce_scatter` implementations with explicit CUDA stream management. It supports the optional YCCL library for optimized collectives and handles dtype casting and gradient accumulation logic, reducing communication overhead compared to standard NCCL operations.
Contains patches to the `accelerate` library to inject YaFSDP logic into the standard training loop. It modifies the `FullyShardedDataParallelPlugin` to recognize YaFSDP configurations and wraps models accordingly, demonstrating a robust plugin-based integration strategy.