$ show OpenBMB/VoxCPM
VoxCPM is a tokenizer-free, multilingual text-to-speech (TTS) system developed by OpenBMB. It supports voice cloning, voice design, and controllable speech generation across 30 languages and 9 Chinese dialects, outputting 48kHz audio. The architecture is a diffusion-autoregressive model built on MiniCPM-4, comprising AudioVAE, Local Encoder, Text-Semantic LM, Residual Acoustic LM, and Local DiT. The repository provides multiple interfaces: a Gradio web application (app.py) for interactive use, a command-line interface (voxcpm.cli) for batch processing, a fine-tuning web UI (lora_ft_webui.py) for LoRA and full-parameter training, and Python APIs for integration. It also includes utilities for audio enhancement (ZipEnhancer), timestamp alignment, and text normalization. While primarily an application, it contains research-grade model definitions and training pipelines, making it a hybrid project with a strong application focus.
The LLM identified 4 files worth highlighting for reuse:
Implements LoRALinear, a low-rank adaptation layer for linear modules, along with utility functions to apply LoRA to named linear modules in any PyTorch model. This component is self-contained, depends only on torch, and can be extracted and reused in any project requiring parameter-efficient fine-tuning.
Provides the AudioVAE V2 model with a StreamingVAEDecoder that enables real-time, low-latency audio reconstruction from latent representations. The module is isolated and can be reused as a standalone neural audio codec or compression component in other speech generation systems.
Contains text normalization routines using the wetext library, including markdown cleaning and punctuation handling. This is a generic preprocessing utility that can be extracted and applied to any TTS or NLP pipeline requiring consistent text formatting.
Implements word-level and character-level timestamp alignment using the stable-ts algorithm. This module is self-contained and can be reused in any ASR or forced-alignment workflow to produce precise timestamps from audio and transcription.