$ show babysor/MockingBird
MockingBird is a Chinese voice cloning system implementing the SV2TTS (Speaker Verification to Text-to-Speech) framework. It provides a complete pipeline for few-shot voice cloning: a speaker encoder extracts d-vector embeddings from reference audio, a synthesizer (Tacotron with Global Style Tokens or VITS) converts text to mel-spectrograms conditioned on the speaker embedding, and a vocoder (HiFi-GAN, Fre-GAN, or WaveRNN) generates waveforms. It also includes a PPG-based voice conversion path using a Conformer-based PPG extractor and a PPG2Mel converter. The system supports multiple Chinese datasets (aidatatang_200zh, magicdata, aishell3, data_aishell) and provides web UIs (Streamlit/FastAPI), a PyQt5 toolbox, and CLI tools for training and inference. The codebase is a fork of Real-Time-Voice-Cloning adapted for Mandarin with significant additions including VITS synthesizer, multiple vocoder options, emotion extraction via wav2vec2, and a modern web API layer. While the original author has moved to a commercial product (noiz.ai), the repository shows recent commits (2026) and maintains a large community (36k stars).
The LLM identified 10 files worth highlighting for reuse:
Highly optimized Cython implementation of monotonic alignment search (MAS) for attention alignment in TTS models. Computes the maximum path through an attention matrix using dynamic programming. Standalone module with only numpy dependency, easily extractable for any sequence-to-sequence model requiring hard alignment. Used in VITS and Tacotron-style models for duration prediction and attention alignment.
Clean implementation of the Speaker Encoder with Generalized End-to-End (GE2E) loss. Computes speaker embeddings (d-vectors) via LSTM + linear projection with L2 normalization. The similarity_matrix method efficiently computes pairwise cosine similarities between utterances and centroids (both inclusive and exclusive), and the loss function implements the softmax-based GE2E objective with EER calculation. Minimal dependencies (torch, numpy, scipy), no internal coupling.
Well-structured HiFi-GAN implementation with Generator, MultiPeriodDiscriminator, and MultiScaleDiscriminator. The Generator uses transposed convolutions with weight normalization and residual blocks (ResBlock1/2). Discriminators implement multi-period and multi-scale discrimination for improved audio quality. Clean weight_norm handling with remove_weight_norm() for inference. Only depends on torch and local utils.util.
Fre-GAN generator implementing location-variable convolutions with kernel prediction networks. Novel architecture where convolution kernels are predicted from conditioning mel-spectrograms via a KernelPredictor network, enabling frequency-consistent synthesis. Includes ResBlock1/2 with dilated convolutions and conditional upsampling paths. Demonstrates advanced GAN vocoder techniques beyond standard HiFi-GAN.
Full-featured Conformer encoder for PPG (phonetic posteriorgram) extraction. Supports multiple attention types (self-attention, relative positional), subsampling strategies (conv2d, vgg2l, linear), positional encodings (absolute, scaled, relative), and positionwise feed-forward variants (linear, conv1d, conv1d-linear). Includes macaron-style FFN, convolution modules, and configurable normalization. Used as the frontend for voice conversion PPG extraction.
Comprehensive F0 processing utilities using pyworld's harvest algorithm. Provides compute_f0, f02lf0, continuous F0 conversion via linear interpolation, log-F0 mean/std normalization for speaker adaptation, and low-pass filtering. Handles voiced/unvoiced flags and continuous log-F0 conversion. Clean, reusable signal processing code with only numpy/scipy/pyworld dependencies.
Mathematically rigorous implementation of rational quadratic spline transforms for normalizing flows (used in VITS). Provides piecewise_rational_quadratic_transform with forward/inverse modes, unconstrained variant with tail bounds, and numerically stable searchsorted/binning logic. Implements the exact transforms from 'Neural Spline Flows' (Durkan et al.). Pure PyTorch, no internal dependencies.
Novel Mixture of Logistics (MOL) attention mechanism for location-relative attention in seq2seq models. Models attention weights as a discretized mixture of logistic distributions, enabling sharp, monotonic alignments with uncertainty modeling. Includes learnable mixture weights, scales, and deltas with proper initialization for different reduction factors. Dropout on mixture weights for regularization. Clean PyTorch implementation.
Flexible hyperparameter management system supporting YAML/JSON loading with dot-notation access (HpsYaml) and nested HParams objects. Handles merging user overrides with defaults, parsing comma-separated CLI overrides, and JSON serialization. Used throughout the codebase for consistent configuration. Standalone, no external dependencies beyond PyYAML.
Opyrator framework core: automatically generates FastAPI endpoints and Streamlit UIs from type-annotated Python functions. Inspects function signatures, extracts Pydantic input/output models, and creates compatible API schemas. Supports form data, file uploads, and custom UI renderers. Generic pattern for turning any typed Python function into a web service with minimal boilerplate.