$ show xiaomi-research/unidrivevla
This repository contains the research code for UniDriveVLA, a unified Vision-Language-Action (VLA) model for autonomous driving. It couples a Qwen3-VL based understanding expert with sparse 3D perception and trajectory planning experts via a Mixture-of-Transformers design. The codebase supports two parallel experiment tracks: nuScenes (real-world open-loop evaluation) and Bench2Drive (CARLA closed-loop simulation). It includes full training pipelines (staged pretraining on driving QA datasets, then fine-tuning on nuScenes/Bench2Drive), evaluation harnesses for detection, motion, planning, and VQA, and a CARLA leaderboard integration for scenario-based testing. The repository is primarily a research artifact—it is not a production application but rather a comprehensive release of model definitions, training scripts, data converters, and evaluation metrics for reproducing the paper's results. Significant code duplication exists between the nuScenes and Bench2Drive directories, and many paths are hardcoded, reducing portability. The code demonstrates strong engineering for research (custom CUDA ops, distributed training with DeepSpeed, modular mmdet3d plugins) but lacks the polish of a reusable library.
The LLM identified 4 files worth highlighting for reuse:
Implements a complete detection evaluation suite (AP, NDS, TP metrics) modeled after nuScenes but adapted for Bench2Drive. The `DetectionMetrics` class is config-driven, serializable, and reusable for any 3D detection benchmark. The `accumulate` function performs per-class matching with interpolation, and the `EvalBoxes` container provides generic box management. This component can be extracted and applied to other detection datasets with minimal changes.
Provides modular utility functions for CARLA-based autonomous driving scenarios: waypoint generation, lane change trajectory computation, junction topology analysis, and obstacle detection. Functions like `generate_target_waypoint_list` and `detect_lane_obstacle` are self-contained and reusable in any CARLA project requiring route following or scenario logic.
Implements a flow-matching planning loss with hybrid waypoint loss (HDP), min-SNR weighting, and lateral/jerk penalties. The loss is decoupled from the model architecture and can be reused for any trajectory prediction task. It demonstrates non-obvious skill in combining flow matching with geometric constraints for autonomous driving planning.
Implements `InstanceBank` and `EgoInstanceBank` for temporal query management in sparse perception models. It handles top-k selection, confidence decay, and anchor projection across coordinate frames. This pattern is reusable for any multi-frame 3D detection or tracking system that uses learned queries, and it is isolated from the rest of the model.