$ show HKUST-SAIL/sail-recon
SAIL-Recon is a feed-forward Transformer-based system for large-scale Structure-from-Motion (SfM) and visual localization. It processes input images or video to estimate camera poses, depth maps, and generate 3D point clouds without iterative optimization. The architecture combines a Vision Transformer backbone with an alternating attention aggregator for temporal modeling and specialized heads for camera, depth, and track predictions. The system includes both command-line and Gradio-based demo interfaces for end-to-end 3D reconstruction, along with evaluation scripts for standard benchmarks (TUM, 7Scenes, Tank&Temple, etc.). While research-oriented, it is packaged as a usable application with clear entry points and modular components.
The LLM identified 4 files worth highlighting for reuse:
A modular Vision Transformer implementation supporting register tokens, layerscale, SwiGLU FFN, and rotary position embeddings. Self-contained with minimal dependencies (torch, einops), making it easily reusable in other ViT-based projects for feature extraction tasks.
Implements an alternating attention mechanism that processes video features by switching between frame-level and global attention. Includes key-value caching for efficient autoregressive inference, demonstrating a non-obvious pattern for temporal aggregation in video understanding that balances computational efficiency with global context modeling.
A Dense Prediction Transformer (DPT) head for depth and confidence prediction. Features multi-scale feature fusion and a custom interpolation function to avoid integer overflow limitations, showcasing careful engineering for robust dense prediction that could be adapted to other regression tasks requiring high-resolution outputs.
Provides essential geometry utilities for 3D reconstruction, including point projection/unprojection, camera pose transformations, and point cloud operations. Self-contained and framework-independent, this module solves recurring problems in SfM/MVS pipelines and could be extracted for use in other 3D vision projects without modification.