FloED is a video inpainting system that leverages optical flow guidance to improve temporal consistency and computational efficiency in diffusion-based video generation. The system employs a dual-branch architecture where one branch estimates and refines optical flow between frames, while the main branch performs the actual inpainting guided by flow-adapted features. Key innovations include a training-free latent interpolation method that accelerates denoising through flow warping, and a flow attention cache mechanism that reduces computational overhead when incorporating motion information. The architecture consists of several interconnected components: a RAFT-based optical flow estimator for computing bidirectional flow between frames, a modified UNet3D architecture incorporating temporal modules and flow-guided attention mechanisms, and a pipeline orchestration system that handles the diffusion process. Input video frames and binary masks are processed through a VAE encoder into latent space, where the UNet iteratively denoises the latents conditioned on text prompts, optical flow estimates, and masked latent representations. The decoded output is then blended with the original frames using the masks to produce the final inpainted video. External dependencies include PyTorch-based deep learning libraries (diffusers, transformers), computer vision tools (OpenCV via decord), and utility packages for configuration management and logging. The system is designed for end-to-end video inpainting tasks, taking as input video frames, corresponding binary masks indicating regions to inpaint, and text prompts describing the desired content, producing coherent inpainted video sequences as output.
This repository provides custom nodes for ComfyUI to enable Deforum-based animation generation. It implements frame-by-frame processing of stable diffusion models with parameter scheduling, allowing users to create animated sequences by evolving prompts, seeds, strength, and other parameters over time. The architecture centers around ComfyUI's node system, with specialized nodes for parameter scheduling, prompt engineering, sampling, latent caching, ControlNet integration, image processing, logic control, video I/O, audio synchronization, and advanced noise generation. A critical execution hijack mechanism modifies ComfyUI's internal processing to handle list operations for frame sequences. The system relies on external libraries like Deforum Studio, ControlNet, RAFT, depth models, and FILM for computer vision techniques.