$ show NevSNev/FloED-main
FloED is a video inpainting system that leverages optical flow guidance to improve temporal consistency and computational efficiency in diffusion-based video generation. The system employs a dual-branch architecture where one branch estimates and refines optical flow between frames, while the main branch performs the actual inpainting guided by flow-adapted features. Key innovations include a training-free latent interpolation method that accelerates denoising through flow warping, and a flow attention cache mechanism that reduces computational overhead when incorporating motion information. The architecture consists of several interconnected components: a RAFT-based optical flow estimator for computing bidirectional flow between frames, a modified UNet3D architecture incorporating temporal modules and flow-guided attention mechanisms, and a pipeline orchestration system that handles the diffusion process. Input video frames and binary masks are processed through a VAE encoder into latent space, where the UNet iteratively denoises the latents conditioned on text prompts, optical flow estimates, and masked latent representations. The decoded output is then blended with the original frames using the masks to produce the final inpainted video. External dependencies include PyTorch-based deep learning libraries (diffusers, transformers), computer vision tools (OpenCV via decord), and utility packages for configuration management and logging. The system is designed for end-to-end video inpainting tasks, taking as input video frames, corresponding binary masks indicating regions to inpaint, and text prompts describing the desired content, producing coherent inpainted video sequences as output.
The LLM identified 3 files worth highlighting for reuse:
Self-contained PyTorch implementation of optical flow visualization that converts flow tensors to RGB images using the standard color wheel approach. Handles both batched and non-batched tensor inputs, applies proper normalization for numerical stability, and manages device placement automatically. This utility can be extracted and reused in any computer vision project requiring optical flow visualization with minimal modification.
Collection of low-level computer vision utilities solving common preprocessing needs in neural network workflows. Includes InputPadder (automatically calculates padding to make dimensions compatible with network requirements, e.g., multiples of 8 for stride-based architectures), bilinear_sampler (provides differentiable sampling with proper coordinate normalization), coords_grid (efficient meshgrid generation for coordinate-based operations), and upflow8 (specialized flow field upsampling by factor of 8). These functions demonstrate clean implementation of fundamental CV operations with minimal dependencies (primarily PyTorch and NumPy).
Core implementation of the flow_warp function for warping images or feature maps according to optical flow fields. Properly handles coordinate system conversion from pixel coordinates to the [-1,1] range required by PyTorch's grid_sample function, supports various interpolation modes (bilinear, nearest) and padding modes (zeros, border, reflection), and includes robust shape validation. This fundamental operation is essential for video processing tasks like optical flow refinement, video stabilization, and motion-compensated filtering, and can be reused in diverse projects with minimal adaptation.