$ show Yangr116/VST
Visual Spatial Tuning (VST) is a research framework for enhancing Vision-Language Models (VLMs) with human-like visuospatial abilities, from perception to reasoning. It implements a two-stage pipeline: supervised fine-tuning (SFT) on 4.1M spatial perception samples (VST-P) followed by reinforcement learning (RL) on 135K reasoning samples (VST-R). The framework extends to Vision-Language-Action (VLA) models for robotic manipulation evaluated on the LIBERO benchmark. The repository contains custom model implementations (Qwen2.5VL, Qwen3VL) with sequence parallelism and FSDP support, data preprocessing scripts, evaluation benchmarks (VSIBench, 3D object detection on SUN RGB-D, VLM benchmarks), and a distributed RL training system (verl) with reward functions and advantage estimation. While primarily a research codebase, it includes production-quality training infrastructure and reusable data utilities. The code is actively maintained with recent commits, but has medium technical drift due to tight version pinning and reliance on an internal ByteDance framework (VeOmni) that is not publicly available, which may hinder full reproducibility.
The LLM identified 5 files worth highlighting for reuse:
Comprehensive 3D bounding box utilities including conversion between coordinate systems (XYZ, UVD, Euler, quaternion), projection, and IoU computation. Reusable across any 3D vision project requiring spatial understanding, with clean function signatures and no tight coupling to the rest of VST.
Discretizes continuous robot actions (e.g., end-effector pose, gripper state) into token IDs for autoregressive prediction. Supports multiple action spaces and normalization. A reusable pattern for any VLA or robot learning project that needs to convert continuous control signals into discrete tokens for language model training.
Collection of generic reward functions for RL training: VQA accuracy, relaxed correctness, ANLS, math answer verification, and multi-choice scoring. Each function is self-contained, parameterized, and can be extracted for use in other RL projects with minimal adaptation.
Converts JSON/JSONL datasets to Parquet format with token counting for packing. Handles images and text, supports multi-processing, and produces a token count column for efficient batching. A reusable data preparation script for any multimodal training pipeline using Parquet-based datasets.
DataProto: a standard data exchange protocol for distributed RL training. Wraps TensorDict, non-tensor arrays, and meta-info, supporting chunking, splitting, concatenation, and distributed operations. Demonstrates clean abstraction for data flow between workers in a distributed system, reusable for any RL framework.