$ show resemble-ai/chatterbox
Chatterbox is a state-of-the-art open-source text-to-speech library by Resemble AI, providing multiple TTS models (Chatterbox, Chatterbox-Multilingual, Chatterbox-Turbo, Chatterbox-Nano) and voice conversion. The architecture is modular: a T3 transformer generates speech tokens from text conditioned on speaker embeddings and optional reference audio; an S3Gen pipeline converts tokens to mel-spectrograms via conditional flow matching (CFM) and then to waveforms via a HiFiGAN vocoder. Voice cloning is achieved through a separate voice encoder (LSTM-based speaker embedding) and S3 tokenizer for prompt conditioning. The library supports 23+ languages with sophisticated multilingual tokenization (Cangjie for Chinese, hiragana for Japanese, diacritics for Hebrew, stress for Russian, Jamo decomposition for Korean). All models include imperceptible neural watermarking via Perth. The codebase is well-structured, with clean separation of concerns, efficient inference (KV caching, CFG, distilled single-step decoder for Turbo/Nano), and pinned dependencies. It is primarily an application library but includes research-grade model implementations.
The LLM identified 5 files worth highlighting for reuse:
The T3 (Token-To-Token) model is a transformer-based autoregressive speech token predictor. It supports custom conditioning via T3Cond (speaker embedding, prompt speech tokens, emotion), classifier-free guidance (CFG), and efficient inference with KV cache. The architecture is modular, using HuggingFace backends (Llama or GPT2) with custom embedding layers and logit heads. The inference method implements a manual generation loop with logit processors (repetition penalty, temperature, top-p, min-p) and CFG combination. This component is reusable as a standalone speech token generator for any TTS system.
S3Token2Wav is the complete speech token-to-waveform pipeline. It combines a CFM-based mel decoder (CausalConditionalCFM) with a HiFiGAN vocoder (HiFTGenerator). The embed_ref method extracts reference audio conditioning (prompt tokens, mel-spectrogram, speaker embedding) in a clean, reusable way. The inference method handles token preprocessing, flow matching with configurable timesteps, and vocoder synthesis with fade-in to reduce artifacts. This component is well-isolated and could be reused with different tokenizers or vocoders.
A robust LSTM-based speaker embedding model adapted from Real-Time-Voice-Cloning. It processes mel-spectrograms through a 3-layer LSTM and projects to a 256-dimensional speaker embedding. The inference method handles variable-length utterances by splitting into overlapping partial windows, computing embeddings per window, and averaging. The embeds_from_wavs method provides a convenient interface for extracting speaker embeddings directly from audio. This component is reusable for any speaker verification or voice cloning task.
A sophisticated multilingual tokenizer that handles language-specific text preprocessing: Chinese characters are converted to Cangjie codes via a custom converter with pkuseg segmentation; Japanese text is normalized to hiragana using pykakasi; Hebrew text gets diacritics via dicta_onnx; Korean syllables are decomposed into Jamo; Russian text receives stress marks. The tokenizer prepends language ID tokens and uses a HuggingFace tokenizer backend. This component demonstrates high craftsmanship in handling diverse writing systems and is reusable for any multilingual TTS or NLP system.
CausalConditionalCFM implements conditional flow matching with classifier-free guidance (CFG) and support for meanflow (distilled single-step) models. The solve_euler method is optimized with pre-allocated tensors for CFG batch duplication, avoiding memory format issues. It supports cosine scheduling and configurable inference CFG rate. The basic_euler method provides a simpler path for meanflow models. This component is a clean, reusable implementation of flow matching for any generative audio task.