$ show zju3dv/Scal3R
Scal3R is a research codebase for CVPR 2026 Highlight paper on large-scale 3D reconstruction from unordered images using test-time training. The core contribution is a novel bidirectional attention mechanism with fast weight updates (TTT layers) that adapts during inference to improve reconstruction quality for large scenes. The repository provides an end-to-end inference pipeline: given a folder of images, it estimates camera poses, depth maps, and produces a colored point cloud. The architecture builds on a DINOv2 backbone and VGGT-based aggregator with custom attention heads, loop detection via SALAD, and pose graph optimization with submap management. The codebase is well-structured with separate modules for data loading, model definition, inference orchestration, and visualization tools. It includes fallback implementations for key components (e.g., Python-only loop solver, simple correlation-based loop detector) to improve portability, though GPU and CUDA are assumed. The repository is primarily a research artifact with clean utilities that could be extracted for other 3D vision projects. Evaluation code and dataset scripts are explicitly marked as not yet released, limiting reproducibility until those are provided.
The LLM identified 6 files worth highlighting for reuse:
Comprehensive camera utility module: functions for quaternion-matrix conversions, SVD orthogonalization, intrinsic matrix construction/adjustment, and EasyVolcap YAML export. Well-documented and self-contained; usable in any 3D vision project for camera parameter handling.
Low-level Test-Time Training operators implementing bidirectional LaCT SwiGLU attention with fast weight updates via Newton-Schulz iterations (Muon optimizer style). Includes weight normalization, minibatch apply, and backward approximations. Highly non-trivial and potentially reusable for other TTT architectures or meta-learning research.
Pairwise alignment of point clouds and camera groups using SE3 or Sim3 with robust weighted least squares (Huber, Cauchy, Geman-McClure). Supports optional refinement with RANSAC iterations. Self-contained and reusable for any point cloud registration or pose graph edge computation task.
Loop detection module with two strategies: SALAD global descriptor matching (state-of-the-art) and a fallback correlation-based method. Handles top-k retrieval, geometric verification, and temporal consistency. Reusable as a standalone loop closure detector for visual SLAM or structure-from-motion pipelines.
Utilities for offloading model inputs, outputs, and intermediate states to disk to reduce GPU memory pressure during large-scale inference. Supports streaming with tqdm progress and diff-based differential writes. Can be adapted for any PyTorch pipeline dealing with large datasets.
Compositing script that renders point clouds and cameras from multiple viewpoints and blends frames into a video. Handles coordinate alignment, alpha composition, and frame ordering. Self-contained and modifiable for generic 3D scene visualization.