This repository implements a real-time speech-to-speech AI pipeline that emulates the OpenAI Realtime API over WebSocket and WebRTC. The system chains Voice Activity Detection (VAD), Speech-to-Text (STT), a Language Model (LLM), and Text-to-Speech (TTS) into a low-latency conversational loop. It supports multiple backends for each stage — including Silero VAD, Parakeet TDT, Whisper, Faster Whisper, MLX Whisper, Paraformer, Qwen3-TTS, Kokoro, ChatTTS, OmniVoice, and OpenAI-compatible APIs — all swappable via a CLI or server configuration. The project includes a fully functional browser demo with a dynamic orb UI, noise gate, tool execution (web search, camera), and daily usage limits. The architecture is built around a pipeline of queue-based handlers (VAD -> STT -> LLM -> TTS) with support for speculative turns, barge-in interruption, and cancellation scopes. The core server is a FastAPI application that exposes a single WebSocket endpoint (`/v1/realtime`) and a WebRTC endpoint (`/v1/realtime/calls`), managing per-connection state via `RealtimeService`. The codebase is structured into clear modules: `pipeline/` for core messages and control flow, `STT/`, `TTS/`, `LLM/`, `VAD/` for backend handlers, `api/openai_realtime/` for the protocol implementation, and `demo/` for the web client. Extensive tests (over 3000 lines for the realtime service alone) cover connection lifecycle, audio streaming, session updates, tool calls, and error handling. While primarily an application, the repo has strong data/research elements due to the integration of multiple state-of-the-art open models (Parakeet-TDT, Qwen3-TTS, Gemma 4, etc.) and the inclusion of speculative turn optimization, smart progressive streaming, and compaction prompts. The code is well-crafted with reusable patterns like `CancelScope`, `SpeculativeTurnTracker`, and `Chat` (conversation history manager). However, the alpha status and many optional dependencies mean the API is still evolving, and some platform-specific code (MLX, CUDA, GGML) may require careful setup.
Dograh is an open-source, self-hostable voice AI platform for building, deploying, and monitoring voice agents. It provides a visual workflow builder (node-based graph editor) for designing call flows, with deep integration across telephony providers (Twilio, Plivo, Vonage, Telnyx, Cloudonix, Vobiz, Asterisk ARI), STT/TTS/LLM models (OpenAI, Deepgram, Google, Azure, Cartesia, ElevenLabs, etc.), and real-time audio pipelines via the Pipecat library. The backend is a FastAPI monolith with PostgreSQL, Redis, and object storage, complemented by separate worker services (ARQ for background tasks) and a campaign orchestrator for bulk outbound calling. The system supports multi-tenancy, call recording, knowledge base/RAG, QA analysis, embeddable chat widgets, and an MCP server for AI-assisted workflow management. SDKs for Python and TypeScript enable programmatic workflow creation and outbound call triggering. While the core is a full-stack application, it includes evaluation frameworks for STT providers and post-call QA analysis, giving it a hybrid character but dominated by production application concerns.