$ show microsoft/Magma
Microsoft Magma is a research codebase for a single large multimodal vision-language-action (VLA) foundation model, as opposed to a general-purpose application. The core is an 8B-parameter model built on LLaMA-3-8B-Instruct with a ConvNeXt vision encoder, capable of understanding images/video and producing both textual responses and discretized robot/UI actions. The repository contains the full training stack: model definitions, data preprocessing, dataset-specific constructors, mixed RLDS/PyTorch data loaders, action tokenization, DeepSpeed training configs, and evaluation hooks for robotics benchmarks like LIBERO and SimplerEnv. It also ships FastAPI servers and demo agents for UI automation and robot trajectory planning, but these are secondary to the model training and data pipeline core, making this a hybrid repo with a dominant data-research character. The data lineage is central to the project. Raw heterogeneous datasets—Open X-Embodiment, Ego4D, EpicKitchen, SeeClick, LLaVA, and the in-house Magma-820K—are normalized into a unified multimodal conversation format via per-dataset Constructor subclasses. Visual point tracks from CoTracker2 are clustered, quantized, and rendered as 'Trace-of-Mark' overlays; bounding boxes and labels are rendered as 'Set-of-Mark' overlays. Continuous robot actions are discretized into token IDs aligned with the LLM vocabulary, allowing a standard causal LM objective to train action prediction alongside visual understanding. The resulting model can be deployed via the included FastAPI server for interactive inference. Reproducing the full pipeline requires access to gated or proprietary datasets, CoTracker2 checkpoints, a custom transformers fork, and significant GPU resources, so end-to-end reproduction is not turnkey, though inference on the released HuggingFace checkpoint is more accessible.
The LLM identified 4 files worth highlighting for reuse:
Encodes continuous robot action vectors into discrete token IDs by binning each action dimension and mapping bins to least-used LLM vocabulary tokens. This cleanly solves the action-as-language problem and is directly reusable in any VLA pipeline that needs continuous trajectories converted to discrete tokens.
Provides a Constructor class hierarchy that turns raw video/trajectory samples into chat-style training examples with system prompts, image tokens, trace overlays, and action placeholder sequences. Dataset-specific subclasses plug in cleanly, making this a modular, reusable pattern for building instruction-tuning datasets from heterogeneous multimodal sources.
Processes CoTracker point tracks into representative future-motion traces using k-means clustering, speed/factor filtering, and spatial quantization. This is the core transformation for Trace-of-Mark auxiliary supervision and can be adapted to any point-track or video-motion dataset without depending on Magma-specific business logic.
Wraps TensorFlow RLDS datasets from Open X-Embodiment into PyTorch interleaved dataset loaders, handling distributed sharding, trajectory chunking, and multi-dataset mixing. It solves the tricky TF-to-PyTorch data boundary and is reusable for any RLDS-based robotics or offline RL dataset collection.