$ show jingyaogong/minimind-v
This repository provides a minimal, from-scratch implementation of a Vision-Language Model (VLM) called MiniMind-V, designed for educational purposes and low-resource training. It ingests the ALLaVA-4V dataset (Parquet format with image bytes and JSON conversations) for pretraining (1.27M caption pairs) and supervised fine-tuning (2.9M mixed instruct/caption/text). The architecture follows a LLaVA-1.5 style: a frozen SigLIP2 visual encoder (256×256, 64 patch tokens) projects via a 2-layer MLP into a MiniMind LLM (with optional Mixture-of-Experts). Training freezes the visual encoder and trains only the projector and selected LLM layers, achieving ~3 RMB / 2 hours on a single RTX 3090. The codebase includes training scripts (pretrain and SFT), evaluation, a Gradio web demo, and model conversion to HuggingFace Transformers format. While primarily a research/educational project, it demonstrates a clean, modular VLM pipeline that can be adapted for other multimodal tasks.
The LLM identified 4 files worth highlighting for reuse:
Defines the MiniMindVLM class, which inherits from MiniMindForCausalLM and adds a vision encoder (SigLIP2) and a 2-layer MLP projector. The architecture is cleanly separated: the visual encoder is frozen, the projector is trainable, and the LLM can be partially frozen (first/last layers). This pattern is reusable for any VLM that needs to adapt a pretrained LLM to vision tasks with minimal compute.
Implements VLMDataset, which loads Parquet files containing image bytes and JSON conversation data. It handles image decoding, chat template processing (system prompt injection, thinking tags), and label masking for autoregressive training. The design is parameterized and could be reused for any multimodal dataset stored in Parquet format with conversational structure.
Provides reusable training utilities: init_vlm_model() for model instantiation with config, vlm_checkpoint() for saving/loading with GPU count adaptation, SkipBatchSampler for skipping corrupted batches, and vlm_collate_fn for dynamic padding. These functions are shared between pretrain and SFT trainers, demonstrating a clean separation of concerns that can be extracted for other VLM training pipelines.
Converts native PyTorch .pth checkpoints to HuggingFace Transformers format, adjusting config keys and state dict names. This is a common need in research projects and the script is self-contained, handling edge cases like MoE layers and vision encoder keys. It can be adapted for any model conversion task between custom and HF formats.