$ show huggingface/speech-to-speech
This repository implements a low-latency, real-time speech-to-speech conversational AI system. It provides an OpenAI Realtime API-compatible server over WebSocket and WebRTC transports, along with a CLI tool for local, talk, and serve modes. The architecture is modular, with separate handler stages for Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM) reasoning, and Text-to-Speech (TTS), connected via typed queues. Multiple backends are supported for each stage, including local models (Whisper, Parakeet, Kokoro, Qwen3-TTS, MLX) and cloud APIs (OpenAI, vLLM, Hugging Face Inference Providers). A demo web application with HF OAuth, rate limiting, and audio worklets is included for browser-based interaction. The system is designed for production use, with emphasis on low latency through speculative turn handling, cancellation scopes, and turn latency instrumentation. Extensive test suites cover unit, integration, and protocol contract tests, including an evaluation harness for Big Bench Audio. While primarily an application, the repo also includes data/research components such as benchmarking scripts and a custom speech evaluation dataset.
The LLM identified 3 files worth highlighting for reuse:
Implements `SpeculativeTurnTracker`, a thread-safe state machine that manages turn revisions for low-latency barge-in. It handles phases (listening, soft_ended, answering, closed), reopen grace windows, pending reopen candidates, and commit/hold gates. Isolated from the rest of the pipeline except for typed events, it demonstrates a non-obvious pattern for speculative execution in real-time speech systems and is directly reusable in other conversational AI architectures.
Provides a generic `BaseHandler[InT, OutT]` abstract class with setup/process/cleanup/on_session_end lifecycle, typed input/output queues, prefetch support, cancellation via `CancelScope`, and internal timing. This pattern cleanly separates pipeline stage concerns and can be extracted for any streaming data processing pipeline (e.g., video, sensor data) with minimal modification. Demonstrates disciplined resource management and thread safety.
Defines `BackendSpec`, `BackendSelection`, and `build_backend_registry` for decoupled handler creation. Backends are registered with lazy imports, capability flags (`streams_audio_chunks`, `supports_llm_proxy`), and CLI-normalized config. This registry pattern allows adding new model backends without modifying pipeline core and is readily adaptable to any plugin-based system requiring dynamic component selection.