Voicebox is a local-first, open-source desktop application for AI voice cloning, text-to-speech (TTS), speech-to-text (STT), and dictation. It provides a full voice I/O stack that runs entirely on the user's machine, supporting multiple TTS engines (Qwen3-TTS, Chatterbox, Kokoro, TADA, LuxTTS, etc.) with zero-shot voice cloning from short audio samples. The application is built as a Tauri v2 desktop shell (Rust) wrapping a React/TypeScript frontend and a Python FastAPI backend with SQLite storage. Key features include global dictation with auto-paste, voice profile management with effects chains, a multi-track story editor, MCP server integration for AI agents, and GPU acceleration (CUDA, ROCm, MLX, DirectML). The architecture follows a service-oriented pattern with a backend factory for pluggable TTS/STT engines, SSE-based progress streaming, and a platform abstraction layer for cross-platform compatibility. The codebase is well-structured with extensive documentation, CI/CD pipelines, and a large community.
This repository implements a real-time speech-to-speech AI pipeline that emulates the OpenAI Realtime API over WebSocket and WebRTC. The system chains Voice Activity Detection (VAD), Speech-to-Text (STT), a Language Model (LLM), and Text-to-Speech (TTS) into a low-latency conversational loop. It supports multiple backends for each stage — including Silero VAD, Parakeet TDT, Whisper, Faster Whisper, MLX Whisper, Paraformer, Qwen3-TTS, Kokoro, ChatTTS, OmniVoice, and OpenAI-compatible APIs — all swappable via a CLI or server configuration. The project includes a fully functional browser demo with a dynamic orb UI, noise gate, tool execution (web search, camera), and daily usage limits. The architecture is built around a pipeline of queue-based handlers (VAD -> STT -> LLM -> TTS) with support for speculative turns, barge-in interruption, and cancellation scopes. The core server is a FastAPI application that exposes a single WebSocket endpoint (`/v1/realtime`) and a WebRTC endpoint (`/v1/realtime/calls`), managing per-connection state via `RealtimeService`. The codebase is structured into clear modules: `pipeline/` for core messages and control flow, `STT/`, `TTS/`, `LLM/`, `VAD/` for backend handlers, `api/openai_realtime/` for the protocol implementation, and `demo/` for the web client. Extensive tests (over 3000 lines for the realtime service alone) cover connection lifecycle, audio streaming, session updates, tool calls, and error handling. While primarily an application, the repo has strong data/research elements due to the integration of multiple state-of-the-art open models (Parakeet-TDT, Qwen3-TTS, Gemma 4, etc.) and the inclusion of speculative turn optimization, smart progressive streaming, and compaction prompts. The code is well-crafted with reusable patterns like `CancelScope`, `SpeculativeTurnTracker`, and `Chat` (conversation history manager). However, the alpha status and many optional dependencies mean the API is still evolving, and some platform-specific code (MLX, CUDA, GGML) may require careful setup.
SenseVoice is a production-grade speech understanding application built around the SenseVoiceSmall encoder-only model. It provides automatic speech recognition (ASR), language identification, speech emotion recognition (SER), and audio event detection (AED) via a non-autoregressive CTC pipeline. The Python codebase includes a FastAPI inference server, Gradio web UI, ONNX/Libtorch export utilities, a long-audio inference script with bounded memory, and a fine-tuning workflow backed by DeepSpeed. The repository also ships an independent C++ runtime under `runtime/llama.cpp/` that ports the same model to GGUF format for CPU/edge deployment without Python. While the core model (`model.py`) is a thin wrapper around the FunASR library and tightly coupled to its conventions, supporting utilities like CTC forced alignment, device resolution, and SER evaluation are well-isolated and reusable. The llama.cpp C++ components (VAD, SenseVoice decoder, and an OpenAI-compatible streaming server) are self-contained, architecturally clean, and validated against the PyTorch reference, making them the highest-value reusable assets. The project is actively maintained with recent commits, thorough tests, and CI/CD for both container builds and the C++ runtime.