$ show jasonppy/VoiceCraft
VoiceCraft is a token infilling neural codec language model for zero-shot speech editing and text-to-speech (TTS) using only a few seconds of reference audio. The system enables users to generate speech in novel voices or edit existing speech (deletion/insertion/substitution) while preserving speaker identity. It processes audio via EnCodec tokenization and models the token sequence with a transformer architecture featuring adaptive layer normalization and delayed pattern prediction for multi-codebook audio representation. The repository provides a complete pipeline from data preparation to deployment. Training scripts (main.py, z_scripts/*.sh) handle distributed training on datasets like Gigaspeech, while inference is accessible through standalone demos (tts_demo.py, speech_editing_demo.py), batch processing scripts (inference_tts_scale.py, inference_speech_editing_scale.py), a Gradio web interface (gradio_app.py), and a Replicate API wrapper (predict.py). Data processing modules (data/tokenizer.py, data/phonemize_encodec_encode_hf.py) manage phoneme extraction and audio tokenization using external tools like phonemizer and audiocraft's EnCodec. Key architectural strengths include reusable components such as the adaptive layer norm transformer encoder (models/modules/transformer.py), the DelayedPatternProvider for EnCodec's interleaved token structure (models/codebooks_patterns.py), and optimized training utilities like ScaledAdam optimizer and length-based batch sampling. The code demonstrates non-obvious skill in handling audio token generation challenges, though it relies on specific versions of external libraries and includes non-commercial use restrictions. Recent activity (commit 2026-05-30) indicates active maintenance with low technical drift due to dependency pinning.
The LLM identified 5 files worth highlighting for reuse:
Implements a custom TransformerEncoder with adaptive layer normalization (AdaLN) for conditional generation. Includes token and positional embeddings, uses xformers for memory-efficient attention, and supports KV-caching for fast inference. This isolated component can be reused in other sequence-to-sequence tasks requiring conditional transformers.
Provides DelayedPatternProvider to manage the interleaved structure of multi-codebook audio tokens (e.g., from EnCodec). Handles delayed pattern prediction and reversion during autoregressive generation, a non-obvious solution essential for coherent audio synthesis. Self-contained with minimal dependencies, extractable for any audio token-based model.
Wraps phonemizer for text-to-phoneme conversion and audiocraft's EnCodec for audio-to-token/token-to-audio conversion. Offers clean, consistent interfaces for text and audio tokenization that decouple the core logic from external library APIs. Highly reusable in speech processing projects with minimal adaptation.
Implements ScaledAdam optimizer (with gradient clipping) and Eden learning rate scheduler. These optimization techniques improve training stability for large models and are packaged as drop-in replacements for standard PyTorch optimizers. Demonstrates non-obvious skill in addressing common training instabilities in transformer-based models.
Contains LengthBasedBatchSampler that groups sequences by similar length to minimize padding in batches. Significantly improves training efficiency for variable-length data (like speech) without requiring bucketing preprocessing. A general-purpose utility applicable to any sequence modeling task with variable input lengths.