$ show boson-ai/higgs-audio
Higgs Audio v2 is an end-to-end multimodal LLM for expressive text-to-speech and audio generation from Boson AI. This repository contains the full v2 codebase: a Llama-3.2-3B backbone augmented with a DualFFN audio adapter, a unified 25 fps audio tokenizer (semantic + acoustic RVQ codes), ChatML-based data collators, and a serving engine with CUDA graph capture and multi-bucket static KV caches. The model supports zero-shot voice cloning, multi-speaker dialogue generation, and emergent capabilities like humming and background music. The README explicitly states that Higgs Audio v3 is released separately and this repo is only needed for v2/v2.5 reference. The code is organized as a Python package (boson_multimodal) with clear separation between model definition, audio processing, data collation, and serving. The serving engine (HiggsAudioServeEngine) provides both synchronous and async streaming generation. The generation loop is a custom implementation on top of HuggingFace's GenerationMixin that interleaves text token sampling with multi-codebook audio token sampling using a delay pattern. Notable engineering: DualFFN architecture (separate audio FFN paths in selected LLM layers), CUDA graph capture for decode, static KV cache buckets of different sizes, repetition-aware sampling (RAS) for audio tokens, and partially-frozen embedding/linear modules for efficient fine-tuning of new audio tokens. Caveats: the codebase pins transformers to <4.47.0 and relies on internal transformers APIs (LlamaDecoderLayer, LLAMA_ATTENTION_CLASSES), making it fragile against library upgrades. Some files contain broken imports (ac.py references a missing binary module; core_vq.py imports an external xcodec package) but these appear to be dead code paths. The README redirects users to v3, so this repo is primarily useful as v2 reference or for reproducing v2 results.
The LLM identified 5 files worth highlighting for reuse:
Implements the delay pattern from 'Simple and Controllable Music Generation' for parallel multi-codebook audio token generation. build_delay_pattern_mask and revert_delay_pattern are self-contained, well-documented functions that transform audio code sequences into a time-shifted format (and back) so all codebooks can be generated autoregressively in a single stream. Directly reusable in any multi-codebook audio language model.
PartiallyFrozenEmbedding and PartiallyFrozenLinear split a pretrained nn.Embedding or nn.Linear into frozen and trainable segments, enabling selective fine-tuning of newly-added vocabulary tokens while keeping base weights frozen. Includes to_unsplit() to merge back. Self-contained, works with ZeRO-2/3, and directly reusable for any vocabulary-extension fine-tuning scenario.
A wrapper that captures a transformer decoder's forward pass into a torch.cuda.CUDAGraph for fast autoregressive decoding. Handles warmup iterations, input/output buffer management, and graph replay with non-blocking copies. A clean, self-contained pattern for any CUDA-based transformer decoding loop that wants to eliminate Python launch overhead.
A collection of language-aware text normalization utilities for TTS pipelines: split_paragraph chunks text by punctuation with token-count limits (zh/en), remove_emoji strips emoji and modifiers, full_to_half_width converts full-width punctuation, spell_out_number verbalizes Arabic numerals, and remove_repeated_punctuations collapses duplicates. Generic, dependency-light, and directly reusable in any TTS or audio preprocessing stack.
A distributed-safe RVQ codebook implementation with memory-efficient k-means initialization (batched torch.cdist, subsampling) and EMA codebook updates synchronized across GPUs via all_reduce and a custom SyncFunction autograd op. Solves the recurring problem of codebook initialization and dead-code expiry in large-scale distributed VQ training. Derived from Meta's AudioCraft code but with meaningful modifications for memory efficiency and multi-GPU correctness.