$ show ZiYang-xie/WorldGen
WorldGen is a Python library and demo application for generating 3D scenes from text prompts or single images in seconds. It orchestrates a pipeline of state-of-the-art models: a fine-tuned FLUX diffusion model to generate 360° panoramic images, DA-2 for monocular depth estimation on equirectangular images, and optionally OneFormer for semantic segmentation and LaMa for inpainting to handle background occlusions. The 3D scene is produced as a Gaussian splatting point cloud (via a custom RGBD-to-Gaussian conversion) or as a textured mesh. An experimental mode uses Apple's Sharp model for higher-quality Gaussian splats by processing cubemap faces. The codebase demonstrates strong engineering practices for a research project: modular pipeline design, use of submodules for dependencies, custom diffusers pipelines with VAE tiling modifications for seamless panorama generation, and a Viser-based interactive viewer. While the core value is the integrated pipeline, several utility modules (equirectangular projection utilities, splat file handling) are cleanly isolated and reusable in other 360° or 3D vision projects.
The LLM identified 2 files worth highlighting for reuse:
Provides robust, self-contained functions for converting between equirectangular and cubemap projections, extracting perspective views from equirectangular images, and computing cubemap face parameters and extrinsics. Uses NamedTuple for clean data structures and supports arbitrary field-of-view and overlapping views. Reusable in any project involving 360° images without modification.
Implements a SplatFile class for representing and saving 3D Gaussian splat data (centers, colors, opacities, covariances) in the standard .ply format used by 3D Gaussian Splatting. The convert_rgbd_to_gs function back-projects equirectangular RGB-D images into Gaussian primitives with analytically computed covariances to cover one pixel in the projection. Self-contained, uses only numpy, torch, pytorch3d, and plyfile, making it reusable for other RGBD-to-Gaussian splatting pipelines.