$ show lipku/LiveTalking
LiveTalking is a real-time interactive streaming digital human engine that synchronizes lip movements with audio generated from text or speech input. It functions as a production-grade web server (Flask + aiohttp) supporting WebRTC, RTMP, and virtual camera output, enabling applications like live streaming, virtual assistants, and conversational avatars. The system integrates a pluggable pipeline: user input → optional LLM (OpenAI-compatible) → TTS (8+ providers including EdgeTTS, CosyVoice, GPT-SoVITS) → lip-sync model inference (Wav2Lip, MuseTalk, Ultralight Digital Human) → video frame compositing → streaming output. It also includes ASR via a local FunASR/SenseVoice WebSocket server and a web admin UI for session management and avatar generation. Architecturally, it follows a plugin-based design using a decorator registry (registry.py) that allows extending TTS, Avatar, and Output modules via registration. The core rendering loop in BaseAvatar manages three concurrent threads (TTS, inference, frame processing) with queue-based data flow. WebRTC integration uses aiortc for peer connections, and the system supports multi-session concurrency with session limits. While primarily an application, it embeds research-derived model code (Whisper, face detection, face parsing) and includes a benchmark script for ASR latency comparison. The codebase has some technical debt: duplicated face detection utilities, stale Dockerfile (Python 3.10/torch 1.12 vs README's newer requirements), and commented-out legacy code. However, the core architecture is solid, and the plugin system and audio utilities are well-crafted and reusable.
The LLM identified 4 files worth highlighting for reuse:
Implements a clean, decorator-based plugin registration system. Modules can register themselves with a category (tts, avatar, output, stt) and name, and the registry creates instances by name with dynamic arguments. The pattern is self-contained (only uses Python standard library) and can be extracted to any Python project needing a simple extensible component factory.
Provides standalone audio utility functions for resampling, channel conversion, and bit-depth adjustment using numpy and soundfile. These are common audio preprocessing tasks needed in many audio/video projects. The functions are pure, well-typed, and independent of the rest of the codebase, making them easily reusable.
Contains helper functions for merging multiple async iterables (merge_async_iters) and wrapping a synchronous generator as an async iterator (async_queue_iter). These solve common problems in async streaming pipelines and are implemented with asyncio primitives only, making them portable to any async Python project.
Defines an abstract base class for audio feature extractors with a clean interface: from_buffer, len, get_stride, get_context_window, and slicing helpers (set_stride, get_stride_iter, get_fill_iter). It also includes a feature buffer manager for sliding window inference. The pattern is generic enough to be reused for any audio feature extraction task requiring fixed-size frames and overlap.