$ show CorentinJ/Real-Time-Voice-Cloning
This is an implementation of SV2TTS (Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis) with a real-time vocoder, structured as a three-stage deep learning pipeline: speaker encoder, Tacotron-based synthesizer, and WaveRNN vocoder. The system clones a voice from a few seconds of reference audio and then synthesizes arbitrary text in that voice. The architecture is a monolithic Python application with CLI and PyQt5 GUI entry points. The encoder module implements a GE2E-loss LSTM speaker encoder trained on VoxCeleb/LibriSpeech. The synthesizer module is a Tacotron variant conditioned on speaker embeddings. The vocoder module is a modified WaveRNN (fatchord version) that generates audio in real-time. All three stages have independent training scripts and inference interfaces. The codebase was a landmark open-source voice cloning project but is now significantly outdated. It pins Python 3.9, PyTorch 1.10, and uses deprecated APIs (scipy.ndimage.morphology.binary_dilation, visdom, webrtcvad). The synthesizer uses a pure-Python instead of CUDA-optimized attention mechanism. The deepmind_version.py vocoder references non-existent utils modules, suggesting incomplete refactoring. The CI pipeline only runs import smoke tests.
The LLM identified 8 files worth highlighting for reuse:
Implements the GE2E speaker encoder with LSTM-based architecture, inclusive/exclusive centroid similarity matrix computation, and EER calculation via ROC curve analysis. The similarity_matrix() method demonstrates an efficient vectorized implementation pattern for computing speaker verification scores that could be extracted for any speaker verification task.
A utility class that provides constrained random sampling from a sequence, guaranteeing each item appears between m//n and ((m-1)//n)+1 times with bounded gap between repetitions. This is a clean, self-contained pattern for any application needing balanced sampling from an iterable without replacement within cycles.
The compute_partial_slices() function implements a sliding window algorithm for splitting variable-length utterances into fixed-length overlapping segments with configurable coverage thresholds. This pattern is reusable for any signal processing pipeline that needs to process long sequences in fixed-size chunks with controlled overlap.
A complete Tacotron implementation with CBHG encoder, LSA (Location-Sensitive Attention) decoder with monotonic accumulated attention, and speaker embedding conditioning via concatenation at each encoder timestep. The LSA attention with cumulative tracking and the CBHG module (convolution bank + highway + GRU) are reusable patterns for sequence-to-sequence models requiring monotonic alignment.
WaveRNN implementation with MelResNet conditioning network, 2D upsampling via Stretch2d, and efficient batched generation with overlap-add crossfade (xfade_and_unfold). The fold_with_overlap/xfade_and_unfold pattern for chunk-based generation with smooth crossfading is a reusable technique for any autoregressive waveform generator needing real-time performance.
Implements discretized mixture of logistic distributions loss and sampling, a technique used in WaveNet/WaveRNN vocoders for raw audio generation. The log_sum_exp numerical stability pattern and the mixture sampling logic are reusable for any generative model outputting discrete audio samples via mixture distributions.
Text-to-sequence conversion pipeline supporting ARPAbet phoneme embedding via curly brace syntax, with extensible cleaner architecture. The pattern of mapping text through cleaners to symbol IDs with special token handling (EOS, padding) is a reusable pattern for any text-to-speech or NLP pipeline requiring text normalization and numericalization.
The split_on_silences() function implements intelligent utterance segmentation based on alignment timestamps, with noise profiling from silence regions using logmmse, and re-attachment of segments that are too short. This pattern of using silence-based segmentation with minimum duration constraints and noise reduction is reusable for preprocessing any speech dataset with alignments.