$ show FACEGOOD/FACEGOOD-Audio2Face
This repository implements an audio-driven facial animation pipeline that maps speech audio to blendshape weights for driving a digital human in Unreal Engine. The core is a deep learning model (Audio2Face) trained on paired audio (WAV) and blendshape weight data (exported from Maya). The training pipeline includes: (1) Linear Predictive Coding (LPC) feature extraction from audio via a C++ DLL, (2) data concatenation and selection of variable blendshape indices, (3) training a custom Keras model with a three-part loss (prediction, motion, emotion regularization), and (4) inference via TensorFlow Lite. A test application demonstrates real-time inference with microphone input, speech recognition (via AISpeech API), and UDP communication to UE4. The repository is primarily a research codebase with a demo application, but lacks dependency pinning, environment definitions, and contains hardcoded paths, making reproduction difficult. The model architecture and loss function are well-structured and potentially reusable.
The LLM identified 4 files worth highlighting for reuse:
Defines the Audio2Face Keras model with three custom sub-models: FormantLayer (conv2d stack), ArticulationLayer (conv2d with emotion concatenation), and OutputLayer (dense with dropout). Also implements a custom loss function combining prediction loss (L_P), motion loss (L_M), and emotion regularization loss (L_R). This component is self-contained, well-structured, and can be extracted for other audio-to-parameter regression tasks.
Concatenates LPC features and blendshape labels from multiple audio clips, selects only variable blendshape indices (based on min-max analysis), and removes half of the zero-frame rows to reduce data imbalance. This data engineering pattern is reusable for any regression task with sparse or zero-heavy labels.
Provides two functions: get_audio_frames (splits raw audio into overlapping frames) and c_lpc (computes LPC coefficients via a C DLL). The code is parameterized (sample rate, frame rate, overlap) and cleanly separated from the model. This pattern can be reused for any audio-to-feature pipeline requiring LPC.
Implements a tfliteInference class that loads a TFLite model, runs inference on batched LPC data, and returns blendshape weights. The class handles model conversion from SavedModel to TFLite if needed. This is a clean, reusable wrapper for TFLite inference in Python.