$ show vlongle/pixie
This repository implements Pixie, a research framework for predicting dense physical material fields (density, Young's modulus, Poisson's ratio, material ID) from 2D image observations of 3D objects. Hydra configs drive a multi-stage pipeline: Objaverse object retrieval, Blender rendering, NeRF-based CLIP feature field training (F3RM), Gaussian Splatting training, voxel grid feature extraction, VLM-based or neural material segmentation, and MPM physics simulation via the vendored PhysGaussian codebase. The pipeline.py entry point orchestrates these stages with caching and reproducibility checks, while render.py produces Blender visualizations of segmented physics fields. Real-scene mode extends the same pipeline to captured datasets using COLMAP dataparsers. The repository also contains the full PixieVerse data curation workflow: class-based Objaverse selection using sentence-transformers, large-scale parallel Blender rendering, VLM-based filtering and critique (Gemini/Claude/GPT/Qwen via the vendored vlmx agent framework), and Streamlit tools for human-in-the-loop label correction and simulation validation. The discrete (semantic segmentation) and continuous (property regression) 3D UNet training lives in the vendored Wavelet-Generation submodule, with shared utilities in pixie/training_utils.py and normalization statistics already computed in normalization_stats/. Architecturally this is a hybrid research/application repository: primarily a configuration-driven pipeline framework with CLI entry points, but also a dataset-generation and model-training workflow. It depends heavily on vendored third-party code (nerfstudio, f3rm, PhysGaussian, Wavelet-Generation, vlmx) and shells out to Blender, CUDA rasterizers, and Hugging Face. Quality is research-grade: disciplined Hydra configs, contextual config snapshots, and skip-if-exists caching coexist with hardcoded user-specific paths, commented-out dead code, and a subprocess-heavy orchestration style. Download scripts for pre-trained models and the PixieVerse dataset are included and functional.
The LLM identified 5 files worth highlighting for reuse:
Self-contained module that takes VLM-generated material property ranges plus Python constraint code, then performs rejection sampling to produce physically plausible material dictionaries. The check_all_constraints/evaluate_constraint pattern is a clean, reusable solution for any domain needing valid parameter draws under hard constraints.
Maps 64^3 network predictions back to world-coordinate PLY point clouds. Handles log-normalization inversion, NeRF-to-world transforms via dataparser transforms, one-hot/argmax material decoding, and typed PLY output. Reusable for any voxel-based prediction pipeline needing spatial output in original scene coordinates.
Aggregates per-object and per-class metrics from distributed inference, computes 90% confidence intervals for segmentation accuracy and mean/SE for regression MSE, and emits structured JSON reports. A solid model-evaluation harness pattern usable beyond this repo.
Streamlit web app for flipping VLM classification labels on rendered object images via checkbox grids, saving corrected JSON with full audit details. Demonstrates a cheap, effective human-in-the-loop dataset curation pattern.
Uploads large datasets to Hugging Face with survivor-object filtering, per-class tar archives, progress reporting, and reusable/resumable archives. Useful for any large-dataset publication workflow involving selective object IDs and multi-terabyte payloads.