$ show OpenBMB/MiniCPM-V
The repository provides a comprehensive ecosystem for the MiniCPM-V and MiniCPM-o series of multimodal large language models (MLLMs). It enables efficient on-device inference and real-time full-duplex multimodal interaction (vision + audio + text) with low latency. The core contributions include the OmniLMM architecture with intra-ViT early compression for visual encoding, mixed 4x/16x visual token compression, and support for streaming audio/video inputs. The system includes evaluation frameworks for multimodal benchmarks (MMBench, Video-MME, OmniDocBench), fine-tuning utilities (LoRA and full model), and deployment tools for various platforms (iOS, Android, HarmonyOS, web) and inference backends (vLLM, SGLang, llama.cpp, Ollama). Key entry points are the model inference code in omnilmm/model/, evaluation scripts in eval_mm/vlmevalkit/, fine-tuning in finetune/, and web demos in web_demos/ demonstrating real-time chat with multimodal inputs.
The LLM identified 5 files worth highlighting for reuse:
Implements specialized evaluation metrics for vision-language tasks: ANLS (Normalized Levenshtein Similarity) for OCR evaluation and relaxed accuracy for numerical answer matching. These metrics handle common variations in model outputs (e.g., punctuation, case, numerical tolerance) and are reusable across different VQA and OCR benchmarks without modification.
Defines the ImageBaseDataset class providing a standardized template for image-based evaluation datasets. It handles common operations like prompt construction, prediction processing, and metric calculation, allowing new datasets to be added by inheriting and overriding minimal methods. This reduces boilerplate and ensures consistent evaluation protocols.
Implements the perception resampler that compresses visual tokens using intra-ViT early compression and mixed 4x/16x strategies. This key innovation reduces visual feature length while preserving information, enabling efficient multimodal fusion. The module is self-contained, depends only on standard ML libraries, and could be adapted for other vision-language models.
Provides reusable image augmentation utilities (RandomAugment, rotate, shear, etc.) and distributed training helpers (all_gather, get_world_size). These components are decoupled from model-specific logic, demonstrate proper encapsulation, and can be extracted for use in other computer vision or multimodal training pipelines.
Encapsulates WebSocket connection management with automatic reconnection, message queuing, and heartbeat mechanisms. This frontend utility enables low-latency bidirectional streaming of multimodal data (audio/video frames, model responses) and abstracts away the complexity of real-time communication, making it reusable in other interactive web applications.