$ show OpenHelix-Team/LLaVA-VLA
The repository implements LLaVA-VLA, a Vision-Language-Action model for robotic manipulation tasks. It extends the LLaVA framework to predict action sequences from visual inputs, proprioceptive state, and language instructions using a diffusion-based action head. The system includes training pipelines for fine-tuning on datasets like CALVIN, inference servers (Flask and Gradio) for real-time robot control, and evaluation tools for benchmarks. The architecture features modular vision encoders, projectors, and resamplers with factory patterns for flexibility, action tokenization for discrete action prediction, and integration with PEFT/DeepSpeed for efficient training.
The LLM identified 4 files worth highlighting for reuse:
Provides a unified interface for discretizing continuous robot actions into tokens (and vice versa) using uniform binning or other strategies. This component is self-contained, depends only on numpy and torch, and can be extracted for use in any project requiring action discretization for language modeling-based action prediction. It demonstrates a non-obvious skill in designing action tokenization schemes that are crucial for treating action prediction as a sequence modeling task in VLAs.
Implements a factory pattern for creating vision encoders (e.g., CLIP, SigLIP, OpenCLIP) with a uniform interface. This module encapsulates the complexity of initializing different vision tower implementations and allows seamless switching between backbones via configuration. It is highly reusable in any multimodal project that requires support for multiple vision encoders without modifying the core model architecture.
Provides a flexible factory for constructing multimodal projectors (linear, MLP, etc.) that align vision features with the language model's embedding space. This component isolates the projector construction logic, enabling easy experimentation with different projection strategies. It is self-contained and can be reused in other multimodal architectures requiring vision-language feature alignment.
Manages conversation templates for various language models (Llama, Vicuna, etc.) and handles the formatting of multi-turn dialogues with special tokens for image inputs. This module abstracts the complexities of prompt construction for different model families and is essential for maintaining consistent input formats during training and inference. It can be extracted for use in any LLM-based application that requires structured conversational prompts with multimodal capabilities.