This repository is a specialized library of distributed orthonormal optimizers for large-scale language model training, implementing the Dion family of algorithms (Dion, Dion2, Dion3, NorMuon, Muon) alongside traditional optimizers (AdamW, Lion). It provides a complete training pipeline including a GPT model, data loaders for FineWeb datasets, and configurable parallelism strategies (FSDP2, TP, DDP). The core innovation lies in communication-efficient orthogonal gradient updates via submatrix selection, Newton-Schulz iterations, and polar decomposition, all wrapped in an async megabatch framework that overlaps collectives with computation. The codebase is well-structured with modular optimizer implementations, extensive tests (including distributed process spawning), and CUDA graph capture support. It is primarily an application library for ML practitioners, but retains a research character through its multiple algorithm variants and academic references. The high engineering quality and active maintenance make it suitable for both production use and further research extensions.
Higgs Audio v2 is an end-to-end multimodal LLM for expressive text-to-speech and audio generation from Boson AI. This repository contains the full v2 codebase: a Llama-3.2-3B backbone augmented with a DualFFN audio adapter, a unified 25 fps audio tokenizer (semantic + acoustic RVQ codes), ChatML-based data collators, and a serving engine with CUDA graph capture and multi-bucket static KV caches. The model supports zero-shot voice cloning, multi-speaker dialogue generation, and emergent capabilities like humming and background music. The README explicitly states that Higgs Audio v3 is released separately and this repo is only needed for v2/v2.5 reference. The code is organized as a Python package (boson_multimodal) with clear separation between model definition, audio processing, data collation, and serving. The serving engine (HiggsAudioServeEngine) provides both synchronous and async streaming generation. The generation loop is a custom implementation on top of HuggingFace's GenerationMixin that interleaves text token sampling with multi-codebook audio token sampling using a delay pattern. Notable engineering: DualFFN architecture (separate audio FFN paths in selected LLM layers), CUDA graph capture for decode, static KV cache buckets of different sizes, repetition-aware sampling (RAS) for audio tokens, and partially-frozen embedding/linear modules for efficient fine-tuning of new audio tokens. Caveats: the codebase pins transformers to <4.47.0 and relies on internal transformers APIs (LlamaDecoderLayer, LLAMA_ATTENTION_CLASSES), making it fragile against library upgrades. Some files contain broken imports (ac.py references a missing binary module; core_vq.py imports an external xcodec package) but these appear to be dead code paths. The README redirects users to v3, so this repo is primarily useful as v2 reference or for reproducing v2 results.