$ show CIntellifusion/GeometryForcing
This repository implements Geometry Forcing, a research framework for video generation with geometric consistency. The core contribution marries video diffusion models (DFoT - Diffusion Forcing Transformer) with 3D representation learning (VGGT) to produce temporally consistent videos conditioned on camera poses. It includes full training pipelines (Hydra + PyTorch Lightning), a suite of video metrics (FVD, FID, LPIPS, VBench), and evaluation scripts for reprojection/revisit error using DROID-SLAM. The codebase is structured as a research template with reusable base classes for algorithms, data modules, experiments, and metric registries. Raw datasets (RealEstate10K, Kinetics-600, Minecraft) are downloaded and preprocessed into video clips with associated camera parameters. Transformations include VAE encoding for latent diffusion, augmentation (flip, reverse), and sliding-window clip extraction. The primary research question is whether video diffusion can be constrained by explicit 3D geometry to reduce inconsistencies in generated videos, measured by geometry-aware metrics. The repo also demonstrates composable sampling via History Guidance and an alignment loss that projects diffusion latents into VGGT feature space. While the code is well-engineered for research (modular design, config-driven, metric registry), it carries typical research code baggage: hardcoded paths, vendored third-party code (DROID-SLAM, VGGT), potential dependency pinning conflicts, and some dead/commented code. Reproducibility is moderate: dependencies are pinned but no environment definitions beyond requirements.txt, and dataset downloads rely on fragile YouTube scraping.
The LLM identified 4 files worth highlighting for reuse:
Central metric aggregator using a SharedVideoMetricModelRegistry that lazily loads and reuses expensive models (I3D, CLIP, DINO, RAFT, AMT, MUSIQ) across multiple video metrics. Demonstrates a clean pattern for managing model lifecycles and caching in evaluation pipelines. Extractable as a standalone video metric harness with minimal adaptation.
Implements History Guidance, a novel composable sampling technique for video diffusion that uses frequency-domain conditioning by partitioning noise levels into segments. The HistoryGuidanceManager orchestrates multiple history modules and step management. This is a self-contained research contribution that could be extracted and applied to other diffusion models with custom guidance interfaces.
Defines CameraPose and Ray classes with utilities for rotation conversions (quaternion, axis-angle, rotation matrix), projection, and Plücker embedding. These are foundational for any 3D vision project dealing with camera trajectories. The classes are pure Python with no internal project dependencies, making them directly extractable for other repos.
Abstract base classes for video datasets (BaseVideoDataset, BaseSimpleVideoDataset, BaseAdvancedVideoDataset) that encapsulate common patterns: video loading, clip generation with sliding windows, metadata caching, and augmentation interfaces. Demonstrates a solid inheritance hierarchy that can be reused for any video dataset with minimal overrides.