This is a large polyglot monorepo for WiFi-based human sensing (WiFi-DensePose) that turns commodity WiFi routers into spatial intelligence systems. It captures Channel State Information (CSI) from ESP32 nodes to detect presence, estimate 17-keypoint body pose, measure breathing and heart rate, and track movement through walls without cameras or wearables. The system integrates deeply with Home Assistant via MQTT and Matter, provides a HomeKit bridge, and exposes multiple surfaces: a Rust backend workspace (v2/crates) with 50+ crates for signal processing, privacy shielding (VEIL/BFLD), pose physics, and home automation; Python bindings (PyO3) for ML training and inference; TypeScript CLI and MCP tools for AI agent integration; ESP32-C6/S3 firmware with adaptive sensing; React Native mobile app; Lit-based dashboard; and Tauri desktop app. The architecture is event-driven with a streaming engine, coherence-gated privacy layers, and deterministic verification via SHA-256 golden files. The codebase is heavily ADR-driven with extensive test coverage including cross-language parity gates. While primarily an application, it also contains substantial research code (benchmarks, physics simulations, SOTA surveys) and is best described as a research-informed production platform.
SpatialLM is a 3D large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. The system provides end-to-end tools for indoor scene understanding, including layout estimation (walls, doors, windows) and 3D object detection (furniture and indoor objects). The architecture consists of a modular Python library with CLI entry points for inference, training, evaluation, and visualization. It combines point cloud encoders (SceneScript or Sonata) with LLM backbones (Llama or Qwen variants) through a projector network. The system handles the complex task of aligning 3D geometric data with language understanding to produce structured outputs representing architectural layouts and object bounding boxes. Key innovations include the use of space-filling curve encodings (Hilbert and Z-order) for positional encoding in the point cloud transformer, a composable transformation pipeline for point cloud preprocessing, and a sophisticated layout representation system that handles the discrete/continuous coordinate transformations required for architectural elements.
Visual Spatial Tuning (VST) is a research framework for enhancing Vision-Language Models (VLMs) with human-like visuospatial abilities, from perception to reasoning. It implements a two-stage pipeline: supervised fine-tuning (SFT) on 4.1M spatial perception samples (VST-P) followed by reinforcement learning (RL) on 135K reasoning samples (VST-R). The framework extends to Vision-Language-Action (VLA) models for robotic manipulation evaluated on the LIBERO benchmark. The repository contains custom model implementations (Qwen2.5VL, Qwen3VL) with sequence parallelism and FSDP support, data preprocessing scripts, evaluation benchmarks (VSIBench, 3D object detection on SUN RGB-D, VLM benchmarks), and a distributed RL training system (verl) with reward functions and advantage estimation. While primarily a research codebase, it includes production-quality training infrastructure and reusable data utilities. The code is actively maintained with recent commits, but has medium technical drift due to tight version pinning and reliance on an internal ByteDance framework (VeOmni) that is not publicly available, which may hinder full reproducibility.