$ show antgroup/echomimic_v2
EchoMimicV2 is an audio-driven semi-body human animation system that generates talking-head or full-body videos from a single reference image, driving audio, and an optional pose sequence. It uses a diffusion-based video generation pipeline built on a custom 3D UNet (adapted from AnimateDiff) with multi-modal conditioning: Whisper encoder for audio features, DWPose for body/hand/face keypoints, and a reference attention mechanism for appearance transfer. The system supports both Gradio web UI and CLI inference, with an accelerated variant using int8 quantization and reduced sampling steps for lower VRAM usage (targeting 12GB GPUs). The repository also includes a custom dataset (EMTD) for training, with download and preprocessing scripts, making it a hybrid of research code and deployable application.
The LLM identified 4 files worth highlighting for reuse:
Implements ReferenceAttentionControl, a write/read bank mechanism that transfers appearance from a reference image to generated video frames via cross-attention. This pattern is reusable for any image-to-video generation task requiring consistent identity or style transfer.
ContextScheduler implements a sliding window inference strategy for generating long videos with limited temporal context. It splits the video into overlapping chunks, processes each with a diffusion model, and blends the outputs. This pattern is reusable for any video generation pipeline that needs to handle arbitrary-length sequences.
Wraps the Whisper encoder to extract frame-level audio features for conditioning. Provides audio2feat and feature2chunks methods that align audio embeddings with video frame timestamps. This modular audio conditioning component can be reused in any audio-driven generation or synchronization task.
Contains motion_sync, a function that transfers facial landmark motion from a driver video to a reference image by aligning and warping keypoints. This pattern is reusable for any face reenactment or expression transfer pipeline that uses landmark-based motion.