$ show fishaudio/fish-speech
Fish Speech is a production-grade multilingual text-to-speech system with a dual-autoregressive transformer architecture (4B parameter slow AR for temporal semantics, 400M fast AR for residual codebooks). It supports real-time voice cloning from short reference audio (10-30s), multi-speaker multi-turn generation, and fine-grained prosody/emotion control via natural language tags. The system is deployed through multiple interfaces: a Gradio web UI, a React-based modern web UI, a production ASGI API server (Kui) with streaming support, and command-line tools for inference and batch processing. Its architecture is modular: a TTSInferenceEngine orchestrates reference loading, LLM inference, and VQ decoding; an RVQ audio codec (10 codebooks, ~21 Hz frame rate) compresses audio; and the dual-AR model is structurally isomorphic to standard LLMs, enabling reuse of inference optimizations like paged KV cache and RadixAttention. The repository also includes training infrastructure (Lightning + Hydra) with LoRA fine-tuning, protobuf-based dataset pipelines, and a comprehensive i18n framework. While primarily a deployable application, it retains strong research character with published papers, benchmark results, and custom model implementations.
The LLM identified 4 files worth highlighting for reuse:
Provides Message and Conversation classes for building multi-turn TTS dialogues with speaker management. Uses a builder-like pattern (with methods like add_user, add_assistant, add_speaker) and Pydantic validation. Self-contained with no internal repo dependencies beyond standard library and Pydantic. Could be extracted as a standalone TTS conversation builder.
Implements bash-style brace expansion (e.g., {a,b}.txt -> a.txt b.txt) as a pure Python utility with no external dependencies. Demonstrates careful handling of nested braces, escape sequences, and edge cases. Fully isolated and reusable in any Python project needing shell-like path expansion.
A minimal i18n framework that loads locale JSON files and provides a gettext-like function with parameter formatting. Includes locale scanning and automatic language detection via system locale. Self-contained (only imports os, json, logging) and easily extracted as a generic i18n utility for any Python application.
A standalone client library for the Fish Speech API server. Supports MsgPack/JSON serialization, streaming playback via pyaudio, and async HTTP with httpx. Demonstrates a clean pattern for interacting with a streaming TTS API, handling audio chunks in real-time. Could be adapted for any similar speech API.