$ show microsoft/VibeVoice
VibeVoice is a production-grade speech AI framework from Microsoft Research, providing both text-to-speech (TTS) and automatic speech recognition (ASR) capabilities. The repository delivers inference and deployment infrastructure for frontier speech models built on a Qwen2.5 LLM backbone with continuous speech tokenizers at 7.5 Hz and a next-token diffusion framework. It includes a vLLM plugin for high-performance serving with OpenAI-compatible APIs, FastAPI/WebSocket streaming ASR, real-time TTS demos, and Gradio-based interactive applications. The codebase is primarily an application deployment toolkit, with embedded research code for model architectures, tokenizers, and diffusion schedulers. It supports long-form multi-speaker TTS (up to 90 minutes), streaming TTS, and single-pass ASR with diarization across multiple languages. The repository also provides LoRA fine-tuning scripts for ASR adaptation and comprehensive demo scripts for both local and cloud deployment.
The LLM identified 4 files worth highlighting for reuse:
Implements streaming-aware convolution layers (SConv1d, SConvTranspose1d) with a generic VibeVoiceTokenizerStreamingCache that maintains state across chunks. This pattern is reusable for any streaming audio tokenizer that requires causal convolutions with stateful caching, enabling low-latency incremental processing without recomputation.
Provides format-agnostic audio loading, conversion, and normalization utilities (load_audio_ffmpeg, convert_audio_to_mp3, extract_audio_from_video, AudioNormalizer). These are self-contained, parameterized functions that can be extracted and reused in any audio processing pipeline, with proper error handling and concurrency control via semaphore.
Implements DPMSolverMultistepScheduler for efficient diffusion sampling with multiple noise schedules (cosine, exponential, Cauchy, Laplace). This is a standalone, well-structured scheduler that can be reused in any diffusion-based generative model, offering configurable steps and solver order.
Defines ChunkGeometry dataclass and streaming format logic for chunked ASR inference. It handles window splitting, chunk reconstruction, and segment merging with robust error recovery. This pattern is reusable for any streaming ASR system that needs to manage chunk boundaries and reconstruct partial transcripts.