$ show antgroup/echomimic_v3
EchoMimicV3 is a unified multi-modal and multi-task human animation model that generates realistic human videos from audio and/or text prompts. Built upon the Wan2.1-Fun-V1.1-1.3B-InP base model, it supports both talking-head and full-body animation with two versions: preview (standard quality) and flash (optimized for speed and lower VRAM). The system provides Gradio-based interactive demos (app.py, app_mm.py) for real-time video generation from user-uploaded media, command-line inference scripts (infer_flash.py, infer_preview.py, run_flash.sh) for batch processing, and deployment options for ComfyUI, ModelScope, and HuggingFace. Core functionality centers on audio-driven lip-sync and facial animation, with optional text guidance for pose/scene control, leveraging diffusion models, transformer architectures, and multimodal encoders (CLIP for images, T5 for text, Wav2Vec2 for audio).
The LLM identified 3 files worth highlighting for reuse:
Implements TeaCache, a training-free caching mechanism that skips redundant transformer computations across timesteps using polynomial coefficients. Significantly accelerates diffusion model inference (reducing steps to 8 in flash mode) without retraining, demonstrating non-obvious optimization skill. Isolated (only depends on torch/numpy) and reusable in any diffusion-based project.
Provides self-contained utilities for latent space conversions (image/video to latent and vice versa), mask handling, and color preservation across frames. Essential for VAE-based video generation pipelines, with functions like latent2vid, vid2latent, and color correction. Zero external coupling beyond torch/numpy, making it safely reusable in other video generation or image processing projects.
Wrapper around RetinaFace for robust facial landmark detection and face isolation. Extracts key facial features (eyes, nose, mouth contour) and generates face masks for animation tasks. Self-contained with clear input/output (image -> landmarks, mask, cropped face). Demonstrates practical integration of a face detection model and is reusable in any face-related computer vision project requiring preprocessing.