$ show neural-maze/realtime-phone-agents-course
This repository is a production-grade application for building real-time voice agents, primarily designed as an educational course but structured for actual deployment. It implements a complete voice call center system capable of receiving and making phone calls via Twilio, transcribing speech using various STT models (Moonshine, Whisper, Faster Whisper), reasoning with LLMs (Groq), searching property data using Superlinked and Qdrant, and generating speech using TTS models (Kokoro, Orpheus). The architecture is a modular monolith built with FastAPI, utilizing Docker for containerization and RunPod for GPU-accelerated model inference. While the repository includes Jupyter notebooks for educational purposes (Lesson 1-4), the core value lies in the `src/realtime_phone_agents` package, which encapsulates the agent logic, avatar system, and infrastructure integrations. The code demonstrates high craftsmanship in handling async audio streaming, managing external API dependencies, and implementing a flexible avatar registry system. It is suitable for engineers looking to build voice AI applications or understand the integration of LLMs with real-time audio pipelines.
The LLM identified 4 files worth highlighting for reuse:
Encapsulates the entire voice agent pipeline (STT -> Agent -> TTS) into a single reusable class. It handles the FastRTC `ReplyOnPause` handler, manages async audio streaming, integrates Opik tracing for observability, and abstracts the complexity of LangChain agent execution. This component is safely reusable for any voice agent project requiring real-time audio interaction.
Implements a robust property search service using the Superlinked framework. It abstracts the complex query logic and handles graceful fallback from Qdrant Cloud to InMemory execution if the cloud service is unavailable. The service pattern is reusable for any application requiring semantic search over structured data with natural language queries.
Provides a clean registry system for managing conversational avatars defined in YAML files. It supports dynamic loading, versioning hooks for prompt tracking (Opik), and case-insensitive lookup. This component demonstrates a reusable pattern for managing multiple AI personas with distinct system prompts and communication styles.
Implements a sophisticated streaming TTS model for Orpheus on RunPod. It handles the conversion of LLM token streams into audio chunks using a background thread and queue pattern, ensuring low-latency audio generation. This demonstrates non-obvious skill in bridging LLM token output with audio synthesis for real-time applications.