$ show MTLab/PE-Field
This repository implements the Positional Encoding Field (PE-Field) technique for single-image novel view synthesis and controllable spatial image editing. It extends 2D positional encodings to structured 3D fields to enable Diffusion Transformers (DiTs) to model geometry directly in 3D space. The system integrates the FLUX.1-Kontext foundation model with MoGe for depth estimation, allowing users to generate novel views from single images or perform spatially controlled edits via azimuth/elevation parameters. The architecture combines custom PE-Field implementation with components from the Hugging Face Diffusers library and MoGe monocular geometry estimation system. Key entry points include infer_viewchanger_single_v2.py for inference and pvd_utils.py for 3D vision utilities. While the research implementation shows promise, it relies heavily on specific external models and libraries, creating potential maintenance challenges as those dependencies evolve.
The LLM identified 2 files worth highlighting for reuse:
Provides reusable utilities for 3D vision tasks including camera trajectory generation, pose interpolation, and point cloud processing specifically designed for novel view synthesis applications. This component encapsulates complex 3D geometry operations that would be non-trivial to implement from scratch, offering a self-contained solution for spatial reasoning in image generation pipelines.
Contains the MoeModel class implementing monocular geometry estimation for depth, normal, and camera prediction. This component integrates DINOv2 vision transformers with advanced techniques like stochastic depth, layer scale, and layer skipping to achieve robust 3D understanding from single images. The modular design allows extraction for use in other applications requiring monocular depth estimation or surface normal prediction.