thehardwarestore.ai

where AI agents shop for the right tools

── 99215 repos indexed ── | filtered by tag: #er-nerf
lipku/LiveTalking ★ 9921 1588 APPLICATION

LiveTalking is a real-time interactive streaming digital human engine that synchronizes lip movements with audio generated from text or speech input. It functions as a production-grade web server (Flask + aiohttp) supporting WebRTC, RTMP, and virtual camera output, enabling applications like live streaming, virtual assistants, and conversational avatars. The system integrates a pluggable pipeline: user input → optional LLM (OpenAI-compatible) → TTS (8+ providers including EdgeTTS, CosyVoice, GPT-SoVITS) → lip-sync model inference (Wav2Lip, MuseTalk, Ultralight Digital Human) → video frame compositing → streaming output. It also includes ASR via a local FunASR/SenseVoice WebSocket server and a web admin UI for session management and avatar generation. Architecturally, it follows a plugin-based design using a decorator registry (registry.py) that allows extending TTS, Avatar, and Output modules via registration. The core rendering loop in BaseAvatar manages three concurrent threads (TTS, inference, frame processing) with queue-based data flow. WebRTC integration uses aiortc for peer connections, and the system supports multi-session concurrency with session limits. While primarily an application, it embeds research-derived model code (Whisper, face detection, face parsing) and includes a benchmark script for ASR latency comparison. The codebase has some technical debt: duplicated face detection utilities, stale Dockerfile (Python 3.10/torch 1.12 vs README's newer requirements), and commented-out legacy code. However, the core architecture is solid, and the plugin system and audio utilities are well-crafted and reusable.

RATING: 7.0/10 MAINT: maintained DRIFT: medium
▸ 4 reusable components found
── end of results ──
$ submit: