FloED is a video inpainting system that leverages optical flow guidance to improve temporal consistency and computational efficiency in diffusion-based video generation. The system employs a dual-branch architecture where one branch estimates and refines optical flow between frames, while the main branch performs the actual inpainting guided by flow-adapted features. Key innovations include a training-free latent interpolation method that accelerates denoising through flow warping, and a flow attention cache mechanism that reduces computational overhead when incorporating motion information. The architecture consists of several interconnected components: a RAFT-based optical flow estimator for computing bidirectional flow between frames, a modified UNet3D architecture incorporating temporal modules and flow-guided attention mechanisms, and a pipeline orchestration system that handles the diffusion process. Input video frames and binary masks are processed through a VAE encoder into latent space, where the UNet iteratively denoises the latents conditioned on text prompts, optical flow estimates, and masked latent representations. The decoded output is then blended with the original frames using the masks to produce the final inpainted video. External dependencies include PyTorch-based deep learning libraries (diffusers, transformers), computer vision tools (OpenCV via decord), and utility packages for configuration management and logging. The system is designed for end-to-end video inpainting tasks, taking as input video frames, corresponding binary masks indicating regions to inpaint, and text prompts describing the desired content, producing coherent inpainted video sequences as output.
VOID is a Netflix research codebase for removing objects from videos while also erasing their physical side effects, such as shadows, reflections, and secondary motion caused by the removed object. The core runtime is a two-pass diffusion inpainting system built on CogVideoX: pass 1 performs coarse object removal guided by a quadmask, and pass 2 refines temporal consistency using warped noise and optical-flow-inspired conditioning. The repo is primarily an application/research library rather than a pure data project: it ships a `videox_fun` Python package with model definitions, custom diffusion pipelines, inference and training scripts, a FastAPI/Ray inference API, Gradio UIs, and a Colab notebook, alongside synthetic data-generation code that uses Blender and Kubric.
StereoCrafter is an application that converts monocular videos to stereoscopic 3D videos using a two-stage diffusion-based pipeline. The first stage estimates depth via DepthCrafter and generates a right-eye view through forward warping. The second stage refines the stereo video using a fine-tuned Stable Video Diffusion model with spatial tiling for high-resolution processing. The system relies on external dependencies including DepthCrafter and Forward-Warp submodules, and requires manual download of model weights. The codebase demonstrates modular design with clear separation between depth estimation, warping, and inpainting stages.