$ account yandex
YaFSDP is a specialized PyTorch library designed to optimize Fully Sharded Data Parallel (FSDP) training for Large Language Models (LLMs). It implements custom sharding strategies that claim up to 20% performance improvements over standard FSDP by reducing communication overhead and optimizing memory usage. The library provides two integration points: a modern composable API via the `fully_shard` function and a legacy-style `YaFSDP` wrapper class that integrates with the `accelerate` ecosystem. It supports advanced features like ragged parameter sharding, CUDA stream-based communication/computation overlap, and optional integration with the Yandex Collective Communication Library (YCCL) for high-performance collectives. The codebase demonstrates high engineering craftsmanship, particularly in its handling of distributed tensor operations and memory management. It utilizes custom tensor subclasses (`RaggedShardDTensor`) to manage non-uniform parameter sizes within sharded groups and implements low-level CUDA stream synchronization to hide communication latency. However, the library relies on internal PyTorch APIs (e.g., `torch._C`, `torch.distributed.fsdp.wrap._...`) and patches external libraries like `accelerate` and `transformers`, which introduces maintenance risks as upstream frameworks evolve.