$ show hacksider/Deep-Live-Cam
Deep-Live-Cam is a real-time face swapping desktop application that uses ONNX Runtime with insightface models to swap faces in live webcam feeds or pre-recorded images/videos. The core pipeline involves face detection (RetinaFace), face recognition (ArcFace), and a face swap model (inswapper_128), with optional face enhancement via GFPGAN or GPEN-BFR ONNX models. The application provides a PySide6 GUI for selecting source/target faces, toggling options (mouth mask, many faces, Poisson blending, sharpening, temporal interpolation), and managing camera selection. It supports multiple execution providers: CUDA (NVIDIA), CoreML (Apple Silicon), DirectML (AMD/Intel), OpenVINO (Intel), and CPU fallback. Video processing can run in-memory via FFmpeg pipes (avoiding disk I/O) or via traditional frame extraction to PNG files. The architecture is modular: frame processors (face swapper, enhancers) follow a common interface, and the codebase includes optimizations for Apple Silicon (CoreML subgraph optimization, Pad decomposition, Shape/Gather folding) and CUDA (IO binding, CUDA graphs). The project also includes a benchmark pipeline and unit tests for core components.
The LLM identified 6 files worth highlighting for reuse:
Performs ONNX graph transformations to improve CoreML execution on Apple Silicon: folds Shape→Gather chains into constants when input size is known, decomposes Pad(reflect) into Slice+Concat (ORT limitation), decomposes Split into Slice pairs, and widens scalar Gather indices to 1D + Squeeze (ORT bug workaround). All passes cache the optimized model on disk. This module is self-contained and can be extracted for any ONNX model targeting CoreML.
Provides drop-in GPU-accelerated replacements for common OpenCV functions (GaussianBlur, resize, cvtColor, flip, addWeighted, sharpen) using cv2.cuda.GpuMat when CUDA is available, falling back to CPU otherwise. Clean interface with typed wrappers, enabling transparent GPU acceleration for image processing pipelines without conditional logic in client code.
Handles downloading ONNX models from Hugging Face with resume support, progress bars, SSL verification via certifi, and size validation. Thread-safe per-model locks prevent concurrent downloads. Also manages insightface model packs (buffalo_l) by downloading individual components to the correct ~/.insightface directory. This is a reusable utility for any Python project needing model downloading with resilience.
Shared utilities for ONNX-based face enhancement models (GPEN, GFPGAN). Provides session creation with optimized provider config (CUDA IO binding, CoreML options, OpenVINO AUTO), warmup inference, pre/post processing (BGR→RGB normalization, affine alignment based on 5-point landmarks), and a single-face enhance function with feathered edge blending. Includes a threading semaphore to limit concurrent ONNX calls and prevent VRAM exhaustion. Reusable for any face restoration ONNX model.
Provides imread_unicode() and imwrite_unicode() as drop-in replacements for cv2.imread/imwrite that handle non-ASCII file paths on Windows (where OpenCV's ANSI code page limitation causes silent failures). Reads/writes via numpy's unicode-aware file I/O and cv2.imdecode/imencode. Simple, correct, and reusable in any OpenCV project that deals with international file names.
Clusters face embeddings using KMeans with automatic optimal k selection via the elbow method (largest inertia drop). Provides find_cluster_centroids() for video face analysis and find_closest_centroid() for matching detected faces to clusters. Self-contained and reusable for any embedding clustering task.