SpatialLM is a 3D large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. The system provides end-to-end tools for indoor scene understanding, including layout estimation (walls, doors, windows) and 3D object detection (furniture and indoor objects). The architecture consists of a modular Python library with CLI entry points for inference, training, evaluation, and visualization. It combines point cloud encoders (SceneScript or Sonata) with LLM backbones (Llama or Qwen variants) through a projector network. The system handles the complex task of aligning 3D geometric data with language understanding to produce structured outputs representing architectural layouts and object bounding boxes. Key innovations include the use of space-filling curve encodings (Hilbert and Z-order) for positional encoding in the point cloud transformer, a composable transformation pipeline for point cloud preprocessing, and a sophisticated layout representation system that handles the discrete/continuous coordinate transformations required for architectural elements.
Visual Spatial Tuning (VST) is a research framework for enhancing Vision-Language Models (VLMs) with human-like visuospatial abilities, from perception to reasoning. It implements a two-stage pipeline: supervised fine-tuning (SFT) on 4.1M spatial perception samples (VST-P) followed by reinforcement learning (RL) on 135K reasoning samples (VST-R). The framework extends to Vision-Language-Action (VLA) models for robotic manipulation evaluated on the LIBERO benchmark. The repository contains custom model implementations (Qwen2.5VL, Qwen3VL) with sequence parallelism and FSDP support, data preprocessing scripts, evaluation benchmarks (VSIBench, 3D object detection on SUN RGB-D, VLM benchmarks), and a distributed RL training system (verl) with reward functions and advantage estimation. While primarily a research codebase, it includes production-quality training infrastructure and reusable data utilities. The code is actively maintained with recent commits, but has medium technical drift due to tight version pinning and reliance on an internal ByteDance framework (VeOmni) that is not publicly available, which may hinder full reproducibility.
The AVOD repository implements the Aggregate View Object Detection network for 3D object detection from LiDAR and camera data, primarily targeting the KITTI benchmark. It provides a complete pipeline for training, evaluation, and inference, including data loading, model definition (AVOD and RPN variants), feature extraction from bird's-eye view and image modalities, anchor-based proposal generation, and post-processing. The system is structured as a modular Python package with protocol buffer-based configuration. Key architectural features include a multi-sensor fusion approach that combines BEV features from LiDAR data with RGB image features through fully connected layers, and a region proposal network for generating object proposals. The codebase demonstrates strong modularity in certain components such as box encoding/decoding utilities, anchor generators, and feature extractors, which could be reused in other 3D detection projects with minimal modification. However, the core model training and evaluation scripts are tightly coupled to the KITTI dataset format and TensorFlow 1.x specifics. The repository shows signs of technical drift due to its reliance on outdated dependencies (TensorFlow 1.3.0, Python 3.5) and deprecated APIs. While the last commit indicates recent activity, the deep learning framework used is no longer current, which may pose challenges for maintenance and extension. Despite this, the implementation includes well-crafted, reusable geometric utilities and modular design patterns that reflect solid engineering practices for a research codebase.