LiveTalking is a real-time interactive streaming digital human engine that synchronizes lip movements with audio generated from text or speech input. It functions as a production-grade web server (Flask + aiohttp) supporting WebRTC, RTMP, and virtual camera output, enabling applications like live streaming, virtual assistants, and conversational avatars. The system integrates a pluggable pipeline: user input → optional LLM (OpenAI-compatible) → TTS (8+ providers including EdgeTTS, CosyVoice, GPT-SoVITS) → lip-sync model inference (Wav2Lip, MuseTalk, Ultralight Digital Human) → video frame compositing → streaming output. It also includes ASR via a local FunASR/SenseVoice WebSocket server and a web admin UI for session management and avatar generation. Architecturally, it follows a plugin-based design using a decorator registry (registry.py) that allows extending TTS, Avatar, and Output modules via registration. The core rendering loop in BaseAvatar manages three concurrent threads (TTS, inference, frame processing) with queue-based data flow. WebRTC integration uses aiortc for peer connections, and the system supports multi-session concurrency with session limits. While primarily an application, it embeds research-derived model code (Whisper, face detection, face parsing) and includes a benchmark script for ASR latency comparison. The codebase has some technical debt: duplicated face detection utilities, stale Dockerfile (Python 3.10/torch 1.12 vs README's newer requirements), and commented-out legacy code. However, the core architecture is solid, and the plugin system and audio utilities are well-crafted and reusable.
This repository is a curated awesome-list of research papers, projects, datasets, and industry demos related to digital humans: 3D avatar generation and animation, clothed human reconstruction, head avatars, human motion generation, virtual try-on, cloth simulation, and related topics. It contains only a README.md plus LICENSE; there is no application code, no executable pipeline, no tests, no CI, and no dependency manifests. Despite the data_research classification, it is best understood as a manually maintained literature index rather than a data pipeline or software project.
This repository is a mobile SDK for real-time interactive AI avatars, primarily targeting Android and iOS platforms. It provides on-device inference for speech recognition (via WeNet) and face/body segmentation (via ncnn-based U-Net models), enabling low-latency lip-sync and avatar animation driven by audio input. The SDK is structured as a C++ core with platform-specific wrappers (Java for Android, Objective-C for iOS), and includes bundled third-party libraries (ncnn, OpenCV, FFmpeg, ONNX Runtime) for computer vision, audio processing, and neural network inference. In production, the system ingests 16kHz PCM audio from a microphone, extracts MFCC features, runs streaming speech recognition, and simultaneously processes the audio through a U-Net model to generate facial animation frames. These frames are rendered using OpenGL ES (Android) or Metal (iOS) with alpha blending for compositing onto a background. The SDK also includes model download, encryption (AES-128-CBC), and resource management utilities. The architecture is monolithic: all core logic lives in a single C++ module (duix-sdk), with platform-specific integration code in the app directories. The demo app in Android provides a fully functional test harness. While the SDK is feature-complete and demonstrates significant engineering effort in mobile AI inference, the codebase exhibits several quality issues: duplicated third-party headers across architectures, hardcoded encryption keys, outdated library versions (OpenCV 4.6.0, ncnn 2022/2023), and minimal inline documentation. The high star count and fork activity suggest a strong community, but the code itself is functional rather than exemplar. The most reusable components are the reference-counted memory allocator (dh_mem.h) and the AES stream decryption wrapper (gaes_stream.h), which can be extracted for general-purpose use.
This repository implements an audio-driven facial animation pipeline that maps speech audio to blendshape weights for driving a digital human in Unreal Engine. The core is a deep learning model (Audio2Face) trained on paired audio (WAV) and blendshape weight data (exported from Maya). The training pipeline includes: (1) Linear Predictive Coding (LPC) feature extraction from audio via a C++ DLL, (2) data concatenation and selection of variable blendshape indices, (3) training a custom Keras model with a three-part loss (prediction, motion, emotion regularization), and (4) inference via TensorFlow Lite. A test application demonstrates real-time inference with microphone input, speech recognition (via AISpeech API), and UDP communication to UE4. The repository is primarily a research codebase with a demo application, but lacks dependency pinning, environment definitions, and contains hardcoded paths, making reproduction difficult. The model architecture and loss function are well-structured and potentially reusable.