$ show knightyxp/VideoCoF
VideoCoF is a unified video editing framework based on diffusion transformers (Wan2.1). It implements a 'See → Reason → Edit' pipeline using chain-of-frames (CoT) reasoning, enabling object removal, addition, swap, and local style transfer. The architecture includes a modular pipeline (`WanPipeline`) with flow matching solvers, VAE, text encoder, and transformer backbone. Inference can be done via a fast inference script (`fast_infer.py`) using 4-step DMD, batch inference via JSON, or through a Gradio UI. Training supports LoRA fine-tuning with DeepSpeed ZeRO-2/3 for 1.3B and 14B models, with custom dataset classes (including CoT and joint image-video). External dependencies include Diffusers, Transformers, Gradio, OpenCV, DeepSpeed, and optional paifuser for optimizations. While primarily an application for video editing, the repo has strong research character with extensive evaluation metrics (CLIP, GPT), configuration files, and experiment scripts. The code is modular in parts but contains experimental features, hardcoded paths, and reliance on an internal optimization package (paifuser) that may limit external reuse. The core utility modules (LoRA, caching, attention, schedulers) are well-crafted and potentially reusable across other diffusion model projects.
The LLM identified 5 files worth highlighting for reuse:
Provides `merge_lora` and `unmerge_lora` functions that can be applied to any Diffusers pipeline for parameter-efficient fine-tuning. The pattern is cleanly isolated and can be extracted for reuse in other projects needing LoRA weight management.
Implements TeaCache, a training-free caching acceleration for diffusion transformers. Precomputed coefficients for Wan2.1 models are included. The technique is independent and can be adapted to other diffusion backends.
Defines a mechanism to select attention backends (FlashAttention 2/3, SageAttention, PyTorch SDP) via environment variable. The pattern of a backend dispatcher is reusable in any transformer-based vision model.
Implements `FlowDPMSolverMultistepScheduler`, a flow-matching DPM solver for diffusion sampling. The scheduler is self-contained and can be plugged into any Diffusers pipeline with flow-matching objectives.
Provides a decorator to skip CFG computation in later diffusion steps, reducing inference cost. The pattern is general and can be applied to any classifier-free guidance pipeline.