whisper.cpp is a high-performance C/C++ inference engine for OpenAI's Whisper automatic speech recognition (ASR) model, with additional support for a wide range of large language models (LLMs) via an integrated llama.cpp fork. The core library is dependency-free and uses the ggml tensor library to run on CPU, GPU (CUDA, Metal, Vulkan, OpenCL, SYCL, etc.), and specialized hardware (Apple Neural Engine, Hexagon DSP, etc.). It provides a clean C API, extensive language bindings (Go, Java, Ruby, JavaScript, Swift, etc.), and a rich set of example applications including a CLI transcriber, real-time microphone streaming, an HTTP server with OpenAI-compatible API, voice-controlled chess, and a talk-llama chatbot. The repository also includes model conversion scripts (PyTorch to ggml, CoreML, OpenVINO), quantization tools, and evaluation benchmarks (LibriSpeech, Earnings21). While primarily an application library, it contains research-oriented features like grammar-constrained decoding, beam search, and experimental model architectures. The codebase is actively maintained with a large community, though the sheer number of hardware backends and model variants introduces some maintenance complexity.
This repository implements a low-latency, real-time speech-to-speech conversational AI system. It provides an OpenAI Realtime API-compatible server over WebSocket and WebRTC transports, along with a CLI tool for local, talk, and serve modes. The architecture is modular, with separate handler stages for Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM) reasoning, and Text-to-Speech (TTS), connected via typed queues. Multiple backends are supported for each stage, including local models (Whisper, Parakeet, Kokoro, Qwen3-TTS, MLX) and cloud APIs (OpenAI, vLLM, Hugging Face Inference Providers). A demo web application with HF OAuth, rate limiting, and audio worklets is included for browser-based interaction. The system is designed for production use, with emphasis on low latency through speculative turn handling, cancellation scopes, and turn latency instrumentation. Extensive test suites cover unit, integration, and protocol contract tests, including an evaluation harness for Big Bench Audio. While primarily an application, the repo also includes data/research components such as benchmarking scripts and a custom speech evaluation dataset.
VibeVoice is a production-grade speech AI framework from Microsoft Research, providing both text-to-speech (TTS) and automatic speech recognition (ASR) capabilities. The repository delivers inference and deployment infrastructure for frontier speech models built on a Qwen2.5 LLM backbone with continuous speech tokenizers at 7.5 Hz and a next-token diffusion framework. It includes a vLLM plugin for high-performance serving with OpenAI-compatible APIs, FastAPI/WebSocket streaming ASR, real-time TTS demos, and Gradio-based interactive applications. The codebase is primarily an application deployment toolkit, with embedded research code for model architectures, tokenizers, and diffusion schedulers. It supports long-form multi-speaker TTS (up to 90 minutes), streaming TTS, and single-pass ASR with diarization across multiple languages. The repository also provides LoRA fine-tuning scripts for ASR adaptation and comprehensive demo scripts for both local and cloud deployment.
ESPnet is an end-to-end speech processing toolkit built on PyTorch, covering a wide range of tasks: automatic speech recognition (ASR), text-to-speech (TTS), speech translation (ST), speech enhancement (SE), speaker recognition, spoken language understanding (SLU), singing voice synthesis (SVS), speech language modeling (SpeechLM), and more. The repository contains both a core Python library (`espnet2/`, `espnet3/`) and a large collection of reproducible recipes (`egs2/`, `egs3/`) for dozens of public corpora and research challenges. The architecture is modular: each task defines its own model, loss, training, and inference via a `Task` class, with abstract base classes for encoders, decoders, frontends, tokenizers, and losses. Configuration is YAML-driven, and the training pipeline uses PyTorch Lightning (in ESPnet3) or a custom trainer (ESPnet2). The system provides CLI tools (`espnet` with subcommands like `transcribe`, `synthesize`, `enhance`, `translate`) and a Python API (`Speech2Text.from_pretrained()`) for inference. It also includes Docker images for deployment and Hugging Face Spaces demos. The repository is hybrid: it is primarily a software framework/library (application) with a strong research character, as evidenced by the extensive recipe collection, benchmark results, and support for state-of-the-art models (Conformer, HuBERT, Whisper, VALL-E). The recipe system follows Kaldi-style data preparation and stage-based pipelines, enabling reproducible experimentation. External dependencies include PyTorch, torchaudio, sentencepiece, and optional components like flash-attn, fairseq, and s3prl. The codebase is actively maintained with CI across multiple OS and Python versions, unit tests, and integration tests.
SenseVoice is a production-grade speech understanding application built around the SenseVoiceSmall encoder-only model. It provides automatic speech recognition (ASR), language identification, speech emotion recognition (SER), and audio event detection (AED) via a non-autoregressive CTC pipeline. The Python codebase includes a FastAPI inference server, Gradio web UI, ONNX/Libtorch export utilities, a long-audio inference script with bounded memory, and a fine-tuning workflow backed by DeepSpeed. The repository also ships an independent C++ runtime under `runtime/llama.cpp/` that ports the same model to GGUF format for CPU/edge deployment without Python. While the core model (`model.py`) is a thin wrapper around the FunASR library and tightly coupled to its conventions, supporting utilities like CTC forced alignment, device resolution, and SER evaluation are well-isolated and reusable. The llama.cpp C++ components (VAD, SenseVoice decoder, and an OpenAI-compatible streaming server) are self-contained, architecturally clean, and validated against the PyTorch reference, making them the highest-value reusable assets. The project is actively maintained with recent commits, thorough tests, and CI/CD for both container builds and the C++ runtime.
FireRedASR is an open-source industrial-grade automatic speech recognition (ASR) system supporting Mandarin, Chinese dialects, and English. It provides two model variants: FireRedASR-AED (Attention-based Encoder-Decoder with Conformer encoder and Transformer decoder) and FireRedASR-LLM (Encoder-Adapter-LLM framework leveraging Qwen2-7B). The codebase includes a CLI tool (`speech2text.py`), a Python API (`FireRedAsr`), feature extraction (fbank with CMVN), tokenization (mixed Chinese/English SentencePiece), and utilities for WER computation. It also includes a complete Triton Inference Server deployment with TensorRT-LLM acceleration for the AED model, with Docker support and benchmarking scripts. The architecture is modular: data preprocessing, model definitions, tokenizers, and inference orchestration are cleanly separated, though the model components are tightly coupled to the specific architectures. The repository is actively maintained with recent commits and demonstrates strong engineering craftsmanship in both the core ASR implementation and the production-grade deployment stack.
Parakeet.cpp is a high-performance speech recognition inference library for NVIDIA's Parakeet model family (TDT-CTC, RNNT, Streaming EOU, Nemotron, Sortformer), written in C++ with C bindings. It provides offline and streaming transcription, speaker diarization, voice activity detection (VAD), beam search with optional n-gram language model fusion, phrase boosting, and timestamp extraction. The library is built on the axiom tensor library with Metal GPU acceleration for Apple Silicon, and supports audio input from WAV, FLAC, MP3, and OGG formats. The codebase includes a CLI tool, a Unix domain socket server example, and a rich set of examples covering all major features. While primarily an application/library for production ASR inference, it also contains research-oriented model conversion scripts (NeMo to safetensors) and a benchmark script comparing with PyTorch inference, giving it a hybrid character. The architecture is cleanly modular, with separate modules for audio I/O, VAD, model encoders/decoders, and decoding algorithms. The C API enables binding to Python, Swift, Rust, and other languages. Documentation and build system are well-maintained, and the repository is actively developed with recent commits.
Violin is an open-source video dubbing application that transcribes speech from any video, translates it into one of 33 languages, synthesizes a native-sounding voice-over using TTS (Cartesia Sonic 3, ElevenLabs, or OpenAI), and remuxes the result into a new video with optional SRT subtitles and in-video Q&A. The system provides three interfaces: a CLI tool (`violin <input> <output> --language <lang>`), a FastAPI web application with a browser UI for upload/download and real-time polling, and a Claude Code skill for invocation from a Claude session. The architecture follows a five-stage pipeline (audio extraction → Whisper transcription → LLM translation → TTS synthesis → video/audio merging with speed alignment) orchestrated by a shared `dub_video` function. Both the CLI and the API worker call the same core pipeline, with parameterized callbacks for progress reporting and cancellation. The API supports bring-your-own-key (BYOK) or server-provided free trial, job persistence via the filesystem, SQLite-based usage statistics, and periodic cleanup of stale jobs and orphaned temp directories. Configuration is YAML-based with deep-merge defaults, supporting pluggable providers per stage. The codebase is well-structured with separate modules for each pipeline stage, reusable cost tracking, and a prompt management system with safe format interpolation. Notable craftsmanship includes the `CostTracker` dataclass for per-job wall-clock and dollar-cost telemetry, the `transcriber.py` segment merging/deduplication logic for Whisper output, and the `api/stats.py` SQLite-backed stats with schema migration. The main weaknesses are the absence of unit tests (no test directory found) and some internal coupling of pipeline components to the global config object. The monolithic SPA frontend is functional but not independently reusable.
Wisp is a local-first, real-time meeting transcription desktop application built with Rust and SvelteKit, using Tauri v2 for cross-platform deployment (macOS Apple Silicon, Windows x64). It captures microphone and system audio, runs on-device automatic speech recognition (ASR) using multiple engines (whisper.cpp, sherpa-onnx, Apple Speech), provides live speaker-diarized transcripts with timestamps, and supports offline semantic search via local ONNX embedding models. The architecture is modular, with pluggable traits for audio sources, ASR engines, denoisers, echo cancellers, and model stores. It also offers optional cloud transcription (OpenAI, Gemini, Groq, etc.) and AI assist features. Although primarily an application, it integrates significant data engineering patterns for streaming audio processing, feature extraction (VAD, diarization), and on-device model management.