The repository provides a comprehensive ecosystem for the MiniCPM-V and MiniCPM-o series of multimodal large language models (MLLMs). It enables efficient on-device inference and real-time full-duplex multimodal interaction (vision + audio + text) with low latency. The core contributions include the OmniLMM architecture with intra-ViT early compression for visual encoding, mixed 4x/16x visual token compression, and support for streaming audio/video inputs. The system includes evaluation frameworks for multimodal benchmarks (MMBench, Video-MME, OmniDocBench), fine-tuning utilities (LoRA and full model), and deployment tools for various platforms (iOS, Android, HarmonyOS, web) and inference backends (vLLM, SGLang, llama.cpp, Ollama). Key entry points are the model inference code in omnilmm/model/, evaluation scripts in eval_mm/vlmevalkit/, fine-tuning in finetune/, and web demos in web_demos/ demonstrating real-time chat with multimodal inputs.
onnx2tf is a production-ready ONNX to TensorFlow/TFLite conversion framework that employs a sophisticated pass-based optimization pipeline. The system converts ONNX models through preprocessing, lowering to an intermediate representation (ModelIR), applying 50+ optimization passes (layout transformation, operator fusion, quantization), and exporting to target formats including TFLite, SavedModel, PyTorch nn.Module, and TorchScript. Its architecture features clean separation between operator implementations, IR transformation passes, and backend-specific exporters, with the flatbuffer_direct backend providing 100x+ faster conversion by bypassing TensorFlow graph construction. The framework includes comprehensive accuracy validation tools, dynamic shape handling, and model surgery capabilities for intermediate layer extraction.