Provides a single `attention()` function that dispatches to multiple backends (torch SDPA, FlashAttention 2/3, SageAttention, xformers) based on a configuration dataclass. Handles GQA, variable-length sequences, and custom masks. This is a self-contained, reusable abstraction that can be extracted into any transformer project needing flexible attention.

Implements `BlockSwapConfig` and `create_offloader()` for CPU offloading of transformer blocks during training/inference. Uses CUDA streams for asynchronous weight transfer. The offloader is generic and can be reused in any large model training pipeline to reduce GPU memory usage.

Provides functions to apply FP8 quantization to linear layers via monkey-patching, supporting scaled and block-wise modes. Includes `load_safetensors_with_fp8_optimization` for loading weights with on-the-fly FP8 conversion. This is a standalone utility that can be adapted for any PyTorch model requiring FP8 memory reduction.

Defines the base `LoRANetwork` class with support for LoRA, LoHa, and LoKr adapters. Handles module replacement, weight merging, optimizer parameter preparation, and max-norm regularization. This is a well-structured, reusable LoRA implementation that can be extracted and used in any PyTorch model fine-tuning project.

Provides common functions for saving and loading latent and text encoder caches as safetensors files. Includes metadata handling, NaN checks, and merging with existing caches. This is a reusable caching layer that can be applied to any diffusion model training pipeline to avoid redundant encoding.