$ show manycore-research/SpatialLM
SpatialLM is a 3D large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. The system provides end-to-end tools for indoor scene understanding, including layout estimation (walls, doors, windows) and 3D object detection (furniture and indoor objects). The architecture consists of a modular Python library with CLI entry points for inference, training, evaluation, and visualization. It combines point cloud encoders (SceneScript or Sonata) with LLM backbones (Llama or Qwen variants) through a projector network. The system handles the complex task of aligning 3D geometric data with language understanding to produce structured outputs representing architectural layouts and object bounding boxes. Key innovations include the use of space-filling curve encodings (Hilbert and Z-order) for positional encoding in the point cloud transformer, a composable transformation pipeline for point cloud preprocessing, and a sophisticated layout representation system that handles the discrete/continuous coordinate transformations required for architectural elements.
The LLM identified 3 files worth highlighting for reuse:
Provides a complete system for representing 3D architectural layouts including walls, doors, windows, and object bounding boxes. Features sophisticated coordinate transformation handling (rotation, translation, scaling), normalization/discretization for LLM consumption, and canonical ordering via sort_key methods. The entity classes are self-contained with clear separation of concerns and minimal dependencies (primarily numpy and scipy), making them highly reusable for other 3D layout or CAD applications.
Implements Hilbert and Z-order space-filling curve encoding and decoding functions for 3D spatial indexing. These mathematical transforms convert 3D coordinates to 1D values while preserving spatial locality, useful for efficient point cloud processing and positional encoding in transformers. The implementations are self-contained, depend only on torch, and demonstrate non-obvious skill in correctly implementing these complex algorithms. Easily extractable for use in other spatial data processing, compression, or indexing applications.
Provides a composable transformation pipeline for point cloud preprocessing using the Registry and Compose patterns. Includes individual transform classes for operations like voxelization, color normalization, jittering, distortion, and sampling that can be chained together flexibly. Demonstrates good software engineering practices in creating modular, reusable preprocessing pipelines where each transform is isolated and follows a clear interface. The pattern is applicable to any point cloud or geometric data processing workflow requiring configurable preprocessing steps.