$ show Robbyant/lingbot-map
This is a research codebase for the Geometric Context Transformer (GCT), a streaming monocular 3D reconstruction model accepted at ECCV 2026. The system processes RGB video frames in real-time (targeting ~20 FPS) to predict camera poses, depth maps, and 3D point clouds using a vision transformer with a novel paged KV-cache causal attention mechanism. It includes a full inference pipeline, interactive and offline renderers (based on Viser and Open3D/CUDA voxelization with octree LOD), and a comprehensive benchmark framework supporting multiple datasets (KTH, Oxford Spires, TUM, 7-Scenes, ETH3D, etc.) with trajectory, depth, and point cloud metrics. The codebase is predominantly application-oriented—providing a deployable model and evaluation harness—but incorporates significant data engineering components such as dataset preprocessing (e.g., Oxford Spires fisheye-to-pinhole conversion with CUDA visibility filtering), standardized BSS artifact storage, and reproducible conda-based environment setup. The core model architecture demonstrates advanced engineering practices: two-stream token layouts, sliding-window KV cache eviction with special-token preservation, 3D RoPE, and modular attention backends (FlashInfer, SDPA, custom causal). The benchmark framework is well-architected with pluggable methods and datasets, resumable execution, and static HTML report generation.
The LLM identified 6 files worth highlighting for reuse:
Implements a two-stream paged KV cache manager for FlashInfer (with SDPA fallback). Separates patch tokens (recyclable, sliding-window eviction) from special tokens (append-only, preserved across evictions). Exposes a plan()/run() pattern that precomputes page tables per layer for efficient multi-head attention, with exact page sizes and zero-padding for FlashAttention-3 compatibility. This is a clean, self-contained component that could be extracted for any streaming transformer requiring efficient KV cache management with mixed token lifetimes.
Defines multiple attention classes (Attention, CausalAttention, FlashInferAttention, SDPAAttention) that share a common interface for causal streaming with KV cache. CausalAttention implements dict-based KV cache with built-in sliding-window eviction and scale-frame retention. FlashInferAttention wraps the paged cache manager. All variants support configurable causal masking and can be swapped at the block level. Demonstrates polymorphic attention design that is reusable across different transformer backends.
Provides point cloud alignment utilities: Umeyama (closed-form similarity transform), ICP refinement with Open3D, voxel downsampling, and alignment evaluation. The functions are standalone (depend only on numpy and optional Open3D) and implement robust registration patterns (centroid subtraction, SVD, RANSAC options) that are readily reusable in any 3D reconstruction or SLAM evaluation pipeline.
Builds a hierarchical octree (via NVIDIA Kaolin SPC) from a point cloud to support efficient LOD rendering. Includes GPU-accelerated construction, level quantization, and dynamic LOD selection during flythrough. This is a self-contained component that solves the common problem of rendering large 3D point clouds with view-dependent detail, and could be reused in any visualization pipeline that uses Kaolin.
Defines CameraPath with keyframe interpolation (linear, bezier), multiple generation functions (follow_path, birdseye, static, pivot), and serialization to/from JSON dictionaries. This is a clean, dependency-light component that could be extracted for any 3D rendering or simulation project requiring scripted camera motion.
Implements the BSS (Benchmark Storage Structure) format: a hierarchical directory layout for predictions, ground truth, and evaluation results with parallel file I/O, metadata handling, and resumable artifact markers. Provides a clean abstraction for large-scale evaluation data that decouples dataset storage from processing logic. The pattern is reusable for any batch evaluation or data pipeline that needs organized artifact management.