$ show MiniMax-AI/VTP
This repository implements VTP (Visual Tokenizer Pre-training), a framework for pre-training visual tokenizers using a combination of contrastive, self-supervised, and reconstruction learning objectives. The system provides a unified model architecture that supports CLIP-style image-text contrastive learning, DINOv2-style self-supervised learning, and image reconstruction. It includes training scripts, evaluation tools for zero-shot classification, linear probing, and reconstruction metrics, as well as integration with LightningDiT for generation evaluation. The codebase features a HuggingFace-compatible model implementation, modular transformer components, and various utility functions for image processing and metric calculation.
The LLM identified 2 files worth highlighting for reuse:
A self-contained implementation of a BPE-based tokenizer for CLIP models. It includes proper handling of special tokens, text cleaning, and BPE encoding/decoding with caching. The implementation has zero internal dependencies and could be extracted for use in any project requiring a CLIP-style tokenizer.
A utility function that converts 3D patch tokens (B, N, C) from vision transformers to 4D format (B, C, H, W) required for convolutional operations. It includes proper validation to ensure the patch count matches the expected spatial dimensions. This solves a common problem in vision transformer codebases with a clean, dependency-free implementation.