thehardwarestore.ai

Voicebox is a local-first, open-source desktop application for AI voice cloning, text-to-speech (TTS), speech-to-text (STT), and dictation. It provides a full voice I/O stack that runs entirely on the user's machine, supporting multiple TTS engines (Qwen3-TTS, Chatterbox, Kokoro, TADA, LuxTTS, etc.) with zero-shot voice cloning from short audio samples. The application is built as a Tauri v2 desktop shell (Rust) wrapping a React/TypeScript frontend and a Python FastAPI backend with SQLite storage. Key features include global dictation with auto-paste, voice profile management with effects chains, a multi-track story editor, MCP server integration for AI agents, and GPU acceleration (CUDA, ROCm, MLX, DirectML). The architecture follows a service-oriented pattern with a backend factory for pluggable TTS/STT engines, SSE-based progress streaming, and a platform abstraction layer for cross-platform compatibility. The codebase is well-structured with extensive documentation, CI/CD pipelines, and a large community.

Moonshine Voice is an open-source AI toolkit for building real-time voice agents and applications with on-device speech-to-text, text-to-speech, and voice cloning capabilities. The system emphasizes privacy, low latency, and offline operation through a cross-platform C++ core with language bindings for Python, Swift, Android, and WebAssembly. It includes optimized models for various platforms (from microcontrollers to desktops), multilingual grapheme-to-phoneme conversion, speaker diarization, and embedding-based phrase matching. The architecture features modular components for audio processing, model inference, and language-specific rules, with examples demonstrating microphone transcription, streaming TTS, agent workflows, and embedded deployments on RP2350 microcontrollers.

Chatterbox is a state-of-the-art open-source text-to-speech library by Resemble AI, providing multiple TTS models (Chatterbox, Chatterbox-Multilingual, Chatterbox-Turbo, Chatterbox-Nano) and voice conversion. The architecture is modular: a T3 transformer generates speech tokens from text conditioned on speaker embeddings and optional reference audio; an S3Gen pipeline converts tokens to mel-spectrograms via conditional flow matching (CFM) and then to waveforms via a HiFiGAN vocoder. Voice cloning is achieved through a separate voice encoder (LSTM-based speaker embedding) and S3 tokenizer for prompt conditioning. The library supports 23+ languages with sophisticated multilingual tokenization (Cangjie for Chinese, hiragana for Japanese, diacritics for Hebrew, stress for Russian, Jamo decomposition for Korean). All models include imperceptible neural watermarking via Perth. The codebase is well-structured, with clean separation of concerns, efficient inference (KV caching, CFG, distilled single-step decoder for Turbo/Nano), and pinned dependencies. It is primarily an application library but includes research-grade model implementations.

VieNeu-TTS is an advanced on-device Vietnamese text-to-speech library with instant voice cloning, supporting bilingual (Vietnamese/English) synthesis. The system provides a Python SDK (`vieneu`) with multiple backends: v3 Turbo (default, 48kHz, ONNX or PyTorch), v3 Nano (lightweight 24kHz, ONNX only), and legacy v1/v2 (deprecated). It includes a Gradio web UI (`vieneu-web`), a FastAPI streaming server, and a browser-based HTML/JS client. The core architecture is a factory pattern (`Vieneu(mode=...)`) that returns the appropriate engine, all inheriting from a shared base class. Inference can run entirely torch-free on CPU via ONNX Runtime, with optional GPU acceleration (CUDA) for batched generation and lower latency. Feature highlights include instant voice cloning from a 3–8 second reference clip using a speaker encoder (ONNX) and denoiser, inline emotion cues (`[cười]`, `[thở dài]`), frame-level streaming, SRT subtitle-to-speech, and LoRA fine-tuning for custom voices (requires ~6GB GPU). The system is deployed via Docker with CPU and GPU profiles, and has CI/CD pipelines for testing and PyPI publishing. The codebase is well-structured with clear separation of concerns, though it carries some legacy weight from older model versions and a few deprecated parameters. Other notable components include a sliding-window repetition penalty, a babble guard heuristic for short utterances, and a voice persistence system for user-defined presets.

Irodori-TTS is a state-of-the-art Japanese text-to-speech system built on rectified flow diffusion over DACVAE audio latents. It provides a complete training and inference stack: `train.py` supports full fine-tuning, LoRA, speaker inversion, and MeanFlow distillation with distributed training and WSD scheduling; `infer.py` offers a rich CLI with CFG, Sway Sampling, duration scaling, and watermarking; `gradio_app.py` and `gradio_app_voicedesign.py` expose interactive UIs for reference-based cloning and caption-conditioned voice design. The architecture is modular (`irodori_tts` package) with attention dispatch (FA3/SDPA), a hybrid Muon+AdamW optimizer, codec abstraction, and caching-friendly `InferenceRuntime`. While highly engineered for production use (INT4/INT8 quantization, safetensors, HuggingFace Hub integration), it remains a research-oriented codebase with experimental sampling algorithms and many YAML configs for different model sizes/versions. The code is clean and well-factored, though `train.py` is a large monolith and external dependencies (DACVAE, silentcipher) are pinned to specific commits, creating ecosystem fragility.

pocket-tts is a production-ready text-to-speech library and application optimized for CPU inference. It provides a Python API, CLI (generate, serve, export-voice), and a FastAPI web server with streaming WAV output. The core architecture uses a FlowLMModel (transformer + flow matching with LSD) and a Mimi neural audio codec, with streaming inference via KV caching. It supports multiple languages (English, French, German, Portuguese, Italian, Spanish), voice cloning from audio prompts, and quantization (int8 dynamic) for efficiency. The repository also includes a full training pipeline (data preparation, alignment, tokenizer training, distributed training, distillation) and evaluation scripts. While primarily an application, the training code adds a research dimension. The codebase is modular and well-documented, with reusable components like stateful streaming modules, quantization utilities, and rotary embeddings. Some technical debt exists (deprecated quantization backend, hardcoded thread settings, multiple config versions), but overall it is a solid, actively maintained project.

YouDub-WebUI is a full-stack web application for video localization (dubbing and subtitling). It provides a user-friendly interface to download videos from YouTube, Bilibili, or local files, then runs a multi-stage ML pipeline: source separation (Demucs), speech-to-text (Whisper), translation via OpenAI Chat Completions, text-to-speech (VoxCPM2), and final audio/video merging with FFmpeg. The backend is a FastAPI server with SQLite persistence, task queue, and background worker; the frontend is a Next.js app with shadcn/ui components. Users can manage tasks, tweak settings (OpenAI model, proxy, concurrency), and view progress in real-time via a serial-polling mechanism. Architecturally, the application follows a monolith with clear separation of concerns: configuration, database, pipeline orchestration, and adapter modules for each ML component. The pipeline is modeled as a series of stages that can be run automatically or manually with step-by-step control. A notable design pattern is the use of adapters (e.g., `backend/app/adapters/`) that wrap third-party libraries and API calls, making the system modular and testable. The frontend uses a custom hook for race-free serial polling to update UI without conflicts. While primarily an application, it has strong research/data components: it bundles heavy ML models (Demucs, Whisper, VoxCPM2) and includes carefully crafted translation prompts for different language pairs (en→zh, ja→zh, zh→en). The pipeline is GPU-optimized with explicit memory management. The repository is hybrid—application with embedded ML research—but the dominant purpose is to serve as a production-ready tool for content creators, validated by a real Bilibili channel with 1M+ followers.

ComfyUI-VoxCPM is a ComfyUI custom node package that provides expressive text-to-speech (TTS) using the VoxCPM model family (v1.x and v2). It supports zero-shot TTS, voice cloning (prompt or reference audio), voice design via text descriptions, LoRA inference, and LoRA fine-tuning. The architecture is a ComfyUI extension with heavy JavaScript frontend, HTTP API routes, and vendored model inference code. The core functionality is implemented in the `src/voxcpm/` directory, which contains model architectures (MiniCPM4, AudioVAE, local DiT, etc.), a standalone CLI, and training utilities. External dependencies include ComfyUI, PyTorch, Hugging Face Hub, and several optional libraries (argbind, datasets, modelscope, etc.). The codebase demonstrates good software engineering practices with clean abstractions, reusable components, and graceful degradation for optional features. However, it is tightly coupled to ComfyUI's ecosystem and vendors upstream model code, which may drift from upstream. The repository also includes a standalone `hf_progress` library for HuggingFace transfer progress tracking, which is a well-isolated, reusable component.

Fish Speech is a production-grade multilingual text-to-speech system with a dual-autoregressive transformer architecture (4B parameter slow AR for temporal semantics, 400M fast AR for residual codebooks). It supports real-time voice cloning from short reference audio (10-30s), multi-speaker multi-turn generation, and fine-grained prosody/emotion control via natural language tags. The system is deployed through multiple interfaces: a Gradio web UI, a React-based modern web UI, a production ASGI API server (Kui) with streaming support, and command-line tools for inference and batch processing. Its architecture is modular: a TTSInferenceEngine orchestrates reference loading, LLM inference, and VQ decoding; an RVQ audio codec (10 codebooks, ~21 Hz frame rate) compresses audio; and the dual-AR model is structurally isomorphic to standard LLMs, enabling reuse of inference optimizations like paged KV cache and RadixAttention. The repository also includes training infrastructure (Lightning + Hydra) with LoRA fine-tuning, protobuf-based dataset pipelines, and a comprehensive i18n framework. While primarily a deployable application, it retains strong research character with published papers, benchmark results, and custom model implementations.

LunaVox is a high-performance, cross-platform text-to-speech inference engine specialized for Qwen3-TTS models. It provides a unified C++ core with Python bindings, offering multiple interfaces: a CLI (`lunavox`), a desktop GUI (customtkinter), an HTTP/WebSocket server (FastAPI), and a Python runtime API. The architecture separates model loading, tokenization, LLM-based talker/predictor generation (via llama.cpp), and neural audio codec decoding (via ONNX Runtime) into modular components. Voice modes include base, voice cloning from reference audio, custom speaker selection, and text-described voice design. The system is designed for low-latency streaming synthesis, with benchmarks showing real-time factors below 0.2 on consumer GPUs. It includes a comprehensive model conversion pipeline from HuggingFace to GGUF/ONNX formats, a build toolchain for compiling the C++ engine, and production-ready serving with Prometheus metrics and auto-batching. The codebase demonstrates strong engineering practices with cross-platform support, structured stats contracts, and bilingual documentation, though it carries some technical debt from vendored GGUF code and platform-specific workarounds.

VoxCPM is a tokenizer-free, multilingual text-to-speech (TTS) system developed by OpenBMB. It supports voice cloning, voice design, and controllable speech generation across 30 languages and 9 Chinese dialects, outputting 48kHz audio. The architecture is a diffusion-autoregressive model built on MiniCPM-4, comprising AudioVAE, Local Encoder, Text-Semantic LM, Residual Acoustic LM, and Local DiT. The repository provides multiple interfaces: a Gradio web application (app.py) for interactive use, a command-line interface (voxcpm.cli) for batch processing, a fine-tuning web UI (lora_ft_webui.py) for LoRA and full-parameter training, and Python APIs for integration. It also includes utilities for audio enhancement (ZipEnhancer), timestamp alignment, and text normalization. While primarily an application, it contains research-grade model definitions and training pipelines, making it a hybrid project with a strong application focus.

MockingBird is a Chinese voice cloning system implementing the SV2TTS (Speaker Verification to Text-to-Speech) framework. It provides a complete pipeline for few-shot voice cloning: a speaker encoder extracts d-vector embeddings from reference audio, a synthesizer (Tacotron with Global Style Tokens or VITS) converts text to mel-spectrograms conditioned on the speaker embedding, and a vocoder (HiFi-GAN, Fre-GAN, or WaveRNN) generates waveforms. It also includes a PPG-based voice conversion path using a Conformer-based PPG extractor and a PPG2Mel converter. The system supports multiple Chinese datasets (aidatatang_200zh, magicdata, aishell3, data_aishell) and provides web UIs (Streamlit/FastAPI), a PyQt5 toolbox, and CLI tools for training and inference. The codebase is a fork of Real-Time-Voice-Cloning adapted for Mandarin with significant additions including VITS synthesizer, multiple vocoder options, emotion extraction via wav2vec2, and a modern web API layer. While the original author has moved to a commercial product (noiz.ai), the repository shows recent commits (2026) and maintains a large community (36k stars).

GPT-SoVITS is a production-grade text-to-speech and voice conversion system with a Gradio-based WebUI and FastAPI REST API. It supports zero-shot voice cloning from a 5-second reference audio and few-shot fine-tuning with 1 minute of data. The system integrates multiple model versions (v1–v4, v2Pro, v2ProPlus) with autoregressive (GPT-style) and non-autoregressive (SoVITS/CFM) components, plus a BigVGAN vocoder. It includes a full pipeline for dataset preparation (voice separation, ASR, text labeling), training (with DDP, LoRA, DPO), and inference (streaming, parallel, ONNX/TorchScript export). The codebase is heavily modular with reusable text frontends for Chinese, Japanese, Korean, Cantonese, and English, as well as audio processing tools (UVR5, BS-Roformer, AP-BWE). Despite its breadth, the code exhibits significant technical debt: version sprawl, hardcoded paths, global state, and experimental hacks. It is actively maintained with frequent bug fixes and feature additions.

Higgs Audio v2 is an end-to-end multimodal LLM for expressive text-to-speech and audio generation from Boson AI. This repository contains the full v2 codebase: a Llama-3.2-3B backbone augmented with a DualFFN audio adapter, a unified 25 fps audio tokenizer (semantic + acoustic RVQ codes), ChatML-based data collators, and a serving engine with CUDA graph capture and multi-bucket static KV caches. The model supports zero-shot voice cloning, multi-speaker dialogue generation, and emergent capabilities like humming and background music. The README explicitly states that Higgs Audio v3 is released separately and this repo is only needed for v2/v2.5 reference. The code is organized as a Python package (boson_multimodal) with clear separation between model definition, audio processing, data collation, and serving. The serving engine (HiggsAudioServeEngine) provides both synchronous and async streaming generation. The generation loop is a custom implementation on top of HuggingFace's GenerationMixin that interleaves text token sampling with multi-codebook audio token sampling using a delay pattern. Notable engineering: DualFFN architecture (separate audio FFN paths in selected LLM layers), CUDA graph capture for decode, static KV cache buckets of different sizes, repetition-aware sampling (RAS) for audio tokens, and partially-frozen embedding/linear modules for efficient fine-tuning of new audio tokens. Caveats: the codebase pins transformers to <4.47.0 and relies on internal transformers APIs (LlamaDecoderLayer, LLAMA_ATTENTION_CLASSES), making it fragile against library upgrades. Some files contain broken imports (ac.py references a missing binary module; core_vq.py imports an external xcodec package) but these appear to be dead code paths. The README redirects users to v3, so this repo is primarily useful as v2 reference or for reproducing v2 results.

This is an implementation of SV2TTS (Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis) with a real-time vocoder, structured as a three-stage deep learning pipeline: speaker encoder, Tacotron-based synthesizer, and WaveRNN vocoder. The system clones a voice from a few seconds of reference audio and then synthesizes arbitrary text in that voice. The architecture is a monolithic Python application with CLI and PyQt5 GUI entry points. The encoder module implements a GE2E-loss LSTM speaker encoder trained on VoxCeleb/LibriSpeech. The synthesizer module is a Tacotron variant conditioned on speaker embeddings. The vocoder module is a modified WaveRNN (fatchord version) that generates audio in real-time. All three stages have independent training scripts and inference interfaces. The codebase was a landmark open-source voice cloning project but is now significantly outdated. It pins Python 3.9, PyTorch 1.10, and uses deprecated APIs (scipy.ndimage.morphology.binary_dilation, visdom, webrtcvad). The synthesizer uses a pure-Python instead of CUDA-optimized attention mechanism. The deepmind_version.py vocoder references non-existent utils modules, suggesting incomplete refactoring. The CI pipeline only runs import smoke tests.

Bert-VITS2 is a multilingual text-to-speech system based on VITS2 (Variational Inference Text-to-Speech with improved Transformer backbone). It supports Chinese, Japanese, and English synthesis with optional emotion conditioning via pre-trained models (WavLM, wav2vec2-emotion, CLAP). The codebase provides a complete pipeline: data preprocessing (resampling, text normalization, BERT feature extraction), multi-speaker training with distributed support, inference through a Gradio web UI or FastAPI server (hiyoriUI.py), and ONNX export for deployment. The architecture uses separate per-language BERT models for text encoding, a flow-based decoder, stochastic duration predictor, and multi-scale discriminators (including a WavLM-based discriminator). The repository includes numerous versioned subdirectories (oldVersion/, onnx_modules/) for backward compatibility with model versions 1.0.1 through 2.4, resulting in significant code duplication and maintenance overhead. The README explicitly states the project is no longer maintained and recommends the successor project Fish-Speech.