$ show open-mmlab/Amphion
Amphion is a comprehensive open-source research toolkit for audio, music, and speech generation, developed by OpenMMLab. It covers multiple generative tasks including text-to-speech (TTS), singing voice conversion (SVC), voice conversion (VC), text-to-audio (TTA), neural audio codecs, and vocoders. The codebase is organized around config-driven experiments (JSON + shell scripts) with modular base classes for trainers, datasets, and inference. It ingests a wide variety of public datasets (LJSpeech, LibriTTS, VCTK, M4Singer, Emilia, etc.) and provides extensive preprocessing pipelines for feature extraction (mel-spectrograms, F0, energy, phonetic alignments, self-supervised features from HuBERT/Whisper). The primary research focus is on high-quality, zero-shot, and controllable generation using architectures like diffusion models, flow matching, GANs, autoregressive transformers, and vector-quantized codecs. The repository also includes evaluation metrics (F0, energy, intelligibility, speaker similarity) and visualization tools (SingVisio). While primarily a research codebase, it includes inference scripts and Gradio demos for practical use. External dependencies include PyTorch, HuggingFace Transformers, WeNet, Descript Audio Codec, and various pretrained models.
The LLM identified 5 files worth highlighting for reuse:
A standalone utility that extracts mel-spectrograms, F0, energy, and content features (e.g., HuBERT, Whisper) from audio waveforms. It is parameterized and can be reused in any audio preprocessing pipeline without modification.
Implements LowPassFilter1d, a critical component for anti-aliasing in neural audio codecs (e.g., BigVGAN, FACodec). It is a small, self-contained module that can be extracted and reused in any waveform generation model.
Provides VectorQuantize and ResidualVectorQuantize implementations with Euclidean codebook, EMA updates, and dead code replacement. This is a clean, reusable quantization module used across multiple codec models (DAC, Amphion Codec).
A clean implementation of F0 Root Mean Square Error metric for evaluating pitch accuracy in speech/singing synthesis. It follows a consistent pattern (extract_f0_rmse) and can be easily extracted for use in other evaluation pipelines.
A grapheme-to-phoneme conversion module supporting multiple languages and backends (espeak, pypinyin, custom lexicons). It is self-contained and reusable for any TTS or voice conversion preprocessing pipeline.