PyTorch3D is a high-performance library for 3D deep learning, providing differentiable rendering, 3D geometry operations, and neural implicit scene representations. The core library offers efficient data structures for meshes, point clouds, and volumes, with packed/padded batching for variable-size inputs. It includes a full differentiable mesh renderer (rasterization, shading, blending), point cloud renderer (including the Pulsar CUDA backend), and an implicit rendering pipeline for NeRF-style models. The library also provides a comprehensive set of 3D operations (KNN, ball query, farthest point sampling, cubify, marching cubes, IoU for 3D boxes, point-mesh distances) and loss functions (chamfer distance, mesh laplacian smoothing, normal consistency). The architecture is modular: cameras, lights, materials, rasterizers, and shaders are composable components. The implicitron subpackage extends this with a configurable framework for novel view synthesis, featuring dataset providers (CO3D, ShapeNet, R2N2), ray samplers, raymarchers, view poolers, and evaluation metrics. Research projects (NeRF, implicitron trainer) demonstrate end-to-end training pipelines with Hydra/OmegaConf configuration. While primarily a library, the repository has strong research characteristics: it includes benchmark scripts, tutorials, and reproduction configs for multiple papers. The codebase is well-tested with over 100 test files, including numerical equivalence tests and benchmarks. Custom CUDA/C++ kernels ensure performance-critical operations are optimized. The library is designed for reuse, with a registry pattern for components, pluggable IO, and a flexible transform system.
SpatialLM is a 3D large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. The system provides end-to-end tools for indoor scene understanding, including layout estimation (walls, doors, windows) and 3D object detection (furniture and indoor objects). The architecture consists of a modular Python library with CLI entry points for inference, training, evaluation, and visualization. It combines point cloud encoders (SceneScript or Sonata) with LLM backbones (Llama or Qwen variants) through a projector network. The system handles the complex task of aligning 3D geometric data with language understanding to produce structured outputs representing architectural layouts and object bounding boxes. Key innovations include the use of space-filling curve encodings (Hilbert and Z-order) for positional encoding in the point cloud transformer, a composable transformation pipeline for point cloud preprocessing, and a sophisticated layout representation system that handles the discrete/continuous coordinate transformations required for architectural elements.
GrowSP is an unsupervised semantic segmentation framework for 3D point clouds that learns from growing superpoints. The system processes raw point cloud data through preparation pipelines, generates initial superpoints using voxelization and region growing, then trains a Feature Pyramid Network (Res16FPN18) to learn discriminative features. During training, superpoints are progressively merged (growing superpoints strategy) to create hierarchical representations, with pseudo-labels generated via clustering of superpoint features. The framework includes specialized dataset classes for S3DIS, ScanNet, and SemanticKITTI, evaluation scripts using Hungarian matching for unsupervised performance measurement, and utility libraries for 3D augmentations and PLY file handling. The architecture follows a modular design with clear separation between data preparation, dataset handling, model definitions, and training/evaluation logic.