The repository provides a comprehensive ecosystem for the MiniCPM-V and MiniCPM-o series of multimodal large language models (MLLMs). It enables efficient on-device inference and real-time full-duplex multimodal interaction (vision + audio + text) with low latency. The core contributions include the OmniLMM architecture with intra-ViT early compression for visual encoding, mixed 4x/16x visual token compression, and support for streaming audio/video inputs. The system includes evaluation frameworks for multimodal benchmarks (MMBench, Video-MME, OmniDocBench), fine-tuning utilities (LoRA and full model), and deployment tools for various platforms (iOS, Android, HarmonyOS, web) and inference backends (vLLM, SGLang, llama.cpp, Ollama). Key entry points are the model inference code in omnilmm/model/, evaluation scripts in eval_mm/vlmevalkit/, fine-tuning in finetune/, and web demos in web_demos/ demonstrating real-time chat with multimodal inputs.
VoxCPM is a tokenizer-free, multilingual text-to-speech (TTS) system developed by OpenBMB. It supports voice cloning, voice design, and controllable speech generation across 30 languages and 9 Chinese dialects, outputting 48kHz audio. The architecture is a diffusion-autoregressive model built on MiniCPM-4, comprising AudioVAE, Local Encoder, Text-Semantic LM, Residual Acoustic LM, and Local DiT. The repository provides multiple interfaces: a Gradio web application (app.py) for interactive use, a command-line interface (voxcpm.cli) for batch processing, a fine-tuning web UI (lora_ft_webui.py) for LoRA and full-parameter training, and Python APIs for integration. It also includes utilities for audio enhancement (ZipEnhancer), timestamp alignment, and text normalization. While primarily an application, it contains research-grade model definitions and training pipelines, making it a hybrid project with a strong application focus.
This repository is a comprehensive collection of tutorials and code examples for deploying, fine-tuning, and integrating open-source large language models (LLMs), with a strong focus on the Chinese LLM ecosystem. It covers over 50 models (Qwen, ChatGLM, DeepSeek, InternLM, etc.) and provides step-by-step guides for setting up FastAPI services, Streamlit web demos, LangChain knowledge base assistants, and LoRA/QLoRA fine-tuning using PEFT. The repository also includes example applications such as role-playing chatbots and digital persona creation. While primarily educational, the code patterns are directly applicable to production scenarios, including model serving with vLLM/SGLang, experiment tracking with SwanLab, and deployment on diverse hardware (NVIDIA, AMD ROCm, Ascend NPU, Apple Silicon). The codebase is highly repetitive across models, with significant copy-paste, but the underlying patterns (fine-tuning pipeline, deployment scaffold, LangChain wrapper) are well-established and reusable.
This repository is a personal cookbook and project showcase centered on the MiniCPM family of large language models (MiniCPM, MiniCPM-V, MiniCPM3). It contains over a dozen distinct application projects, including multi-modal OCR + visual grounding (OCR_VG), cross-modal image-text search, ReAct-style agent planning (agent_auto_plan), MBTI personality role-play, low-memory RAG (4G_memory_rag), AWQ quantization, embedding extraction, SFT/DPO fine-tuning pipelines, and Windows agent demos. Each project includes source code, shell scripts, and accompanying Chinese/English tutorials in markdown format. The codebase is primarily a collection of independent, loosely coupled modules rather than a unified application. It demonstrates practical usage of MiniCPM models for fine-tuning, deployment with vLLM/Ollama/llama.cpp, and integration with LangChain. The quality is functional but inconsistent: many modules contain hardcoded paths, duplicated logic, and lack tests, yet several components (ReAct data generation, image slicing utilities, LangChain wrapper) exhibit sufficient isolation and craftsmanship to be reusable across projects. The repo is actively maintained with a recent commit, but dependency on unmerged forks and version-specific model files poses a medium technical drift risk.