This repository implements a low-latency, real-time speech-to-speech conversational AI system. It provides an OpenAI Realtime API-compatible server over WebSocket and WebRTC transports, along with a CLI tool for local, talk, and serve modes. The architecture is modular, with separate handler stages for Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM) reasoning, and Text-to-Speech (TTS), connected via typed queues. Multiple backends are supported for each stage, including local models (Whisper, Parakeet, Kokoro, Qwen3-TTS, MLX) and cloud APIs (OpenAI, vLLM, Hugging Face Inference Providers). A demo web application with HF OAuth, rate limiting, and audio worklets is included for browser-based interaction. The system is designed for production use, with emphasis on low latency through speculative turn handling, cancellation scopes, and turn latency instrumentation. Extensive test suites cover unit, integration, and protocol contract tests, including an evaluation harness for Big Bench Audio. While primarily an application, the repo also includes data/research components such as benchmarking scripts and a custom speech evaluation dataset.
This repository is a Next.js application demonstrating the use of the OpenAI Realtime API via WebRTC for real-time voice interaction with a 3D solar system scene built in Spline. The application allows users to ask questions about the solar system, moons, the ISS position, and data visualizations, which trigger function calls to update the 3D scene or display charts in the UI. The architecture consists of a frontend built with React and Next.js, featuring components for the 3D scene (using @splinetool/react-spline), controls for managing the WebRTC connection, and a debug log. Two API routes proxy requests to external services: one for fetching the ISS position and another for obtaining ephemeral client secrets from the OpenAI Realtime API. The frontend establishes a WebRTC connection to OpenAI, sends audio input, and processes function call responses to trigger animations in the Spline scene or update the UI with charts. A notable reusable component is the charting system located in `components/charts/chart.tsx`, which provides a set of theme-aware, flexible charting components (Container, Tooltip, Legend) that abstract the complexity of integrating Recharts and Chart.js with Tailwind CSS. This component demonstrates a clean separation of concerns and can be adapted for use in other projects with minimal modification. The rest of the codebase is tailored to the solar system demo but follows modern React and Next.js best practices, including TypeScript strict mode, proper error handling, and modular organization. The application is actively maintained, with recent commits and up-to-date dependencies, and exhibits low technical drift, adhering to current web standards and APIs.