The repository provides a comprehensive ecosystem for the MiniCPM-V and MiniCPM-o series of multimodal large language models (MLLMs). It enables efficient on-device inference and real-time full-duplex multimodal interaction (vision + audio + text) with low latency. The core contributions include the OmniLMM architecture with intra-ViT early compression for visual encoding, mixed 4x/16x visual token compression, and support for streaming audio/video inputs. The system includes evaluation frameworks for multimodal benchmarks (MMBench, Video-MME, OmniDocBench), fine-tuning utilities (LoRA and full model), and deployment tools for various platforms (iOS, Android, HarmonyOS, web) and inference backends (vLLM, SGLang, llama.cpp, Ollama). Key entry points are the model inference code in omnilmm/model/, evaluation scripts in eval_mm/vlmevalkit/, fine-tuning in finetune/, and web demos in web_demos/ demonstrating real-time chat with multimodal inputs.
X-AnyLabeling is a cross-platform desktop application (PyQt6) for AI-assisted annotation of images, videos, and 3D point clouds. It serves as a unified interface for a vast model zoo including YOLO (v5-v26), SAM (1/2/3), Grounding DINO, Florence-2, PP-OCR, and many others, enabling auto-labeling for detection, segmentation, pose estimation, OCR, tracking, and more. The application features a plugin-based model architecture, abstracted inference engines (ONNX, TensorRT, PyTorch), and a comprehensive settings system. It also integrates an Ultralytics YOLO training pipeline and a secure in-app update mechanism. While primarily an application, it has a strong hybrid nature due to its deep integration of research code (vendored Grounding DINO, SAHI) and data pipeline utilities (format converters, COCO evaluators). The codebase demonstrates high craftsmanship in its model abstraction layer and inference engine design, though the sheer volume of integrated models introduces a medium risk of technical drift. It is actively maintained with a large community and extensive test coverage.
This repository provides a comprehensive, production-grade training framework for fine-tuning the Qwen-VL family of vision-language models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5) using HuggingFace Transformers, Liger-Kernel, and DeepSpeed. It supports Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), and classification fine-tuning. The architecture is modular: dataset loaders handle mixed-modality (image/video/text) data with reasoning field support, custom trainer subclasses optimize memory and enable generation-based evaluation, and model wrappers provide sequence classification heads with configurable MLP bridges. Key engineering achievements include monkey-patched vision transformer forward passes for reduced memory usage, robust handling of DeepSpeed ZeRO-3 with dummy vision inputs for text-only batches, and a workaround for PEFT/DeepSpeed compatibility. The codebase is actively maintained, dependencies are pinned, and the project demonstrates strong craftsmanship with several reusable components.
This repository provides a minimal, from-scratch implementation of a Vision-Language Model (VLM) called MiniMind-V, designed for educational purposes and low-resource training. It ingests the ALLaVA-4V dataset (Parquet format with image bytes and JSON conversations) for pretraining (1.27M caption pairs) and supervised fine-tuning (2.9M mixed instruct/caption/text). The architecture follows a LLaVA-1.5 style: a frozen SigLIP2 visual encoder (256×256, 64 patch tokens) projects via a 2-layer MLP into a MiniMind LLM (with optional Mixture-of-Experts). Training freezes the visual encoder and trains only the projector and selected LLM layers, achieving ~3 RMB / 2 hours on a single RTX 3090. The codebase includes training scripts (pretrain and SFT), evaluation, a Gradio web demo, and model conversion to HuggingFace Transformers format. While primarily a research/educational project, it demonstrates a clean, modular VLM pipeline that can be adapted for other multimodal tasks.
Moondream is a lightweight, open-source vision-language model (VLM) library and inference framework. The core of the repository is a PyTorch model (`MoondreamModel`) that comprises a vision encoder, a text decoder (based on a transformer with optional MoE layers), and a region model for spatial grounding. It supports image captioning, visual question answering, object detection, pointing, and gaze detection through a unified API. The library is designed for efficient inference on both GPU and CPU, with support for quantization (via torchao) and LoRA-based fine-tuning. The codebase is cleanly structured into modular components for vision, text, and region processing, each with clear configuration dataclasses. The repository also includes a HuggingFace integration (`HfMoondream`) for easy loading via the transformers library, and several ready-to-use demos (Gradio apps, CLI samples). Beyond the core library, the repo contains a rich set of evaluation scripts (ChartQA, POPE, DocVQA, COCO mAP, etc.) that benchmark the model on standard VLM tasks, and finetuning examples that demonstrate how to adapt the model to custom datasets using reinforcement learning or supervised fine-tuning. Additionally, there are recipe applications (promptable video redaction, content moderation, gaze detection) that showcase practical use cases built on top of Moondream. While the primary character is an application/library, the presence of extensive evaluation and finetuning code gives it a hybrid research-engineering flavor. The code is well-maintained, with active commits and CI, and uses up-to-date dependencies, resulting in low technical drift.
This repository is a large monorepo for a multimodal AI agent platform developed by ByteDance. It includes a desktop Electron application (UI-TARS Desktop) that allows users to control their computer using natural language via vision-language models (UI-TARS, GPT-4o, Claude, etc.). Additionally, it provides a CLI tool (Agent TARS) for running agents in browser/desktop environments, a server for multi-user agent sessions, a web UI for interacting with agents, MCP servers for browser/filesystem/commands, and a comprehensive SDK for building custom agents and operators. The architecture is heavily componentized with multiple packages for agent core, tool-calling engines (native, prompt-engineering, structured-outputs), browser automation (local, remote, sandbox), desktop automation (NutJS, ADB), and UI components. While the codebase demonstrates strong engineering with clean abstractions (event streams, plugin systems, strategy patterns), it also shows signs of rapid growth with some duplication (two agent server implementations, overlapping operator packages) and ongoing migrations. The repository is actively maintained with frequent releases and comprehensive test coverage across unit, integration, and e2e tests.
Pixel-Reasoner implements a two-stage training framework for Vision-Language Models (VLMs) to enable pixel-space reasoning. The system first performs instruction tuning on synthesized reasoning traces to familiarize models with visual operations (zoom-in, select-frame), then applies curiosity-driven reinforcement learning to balance exploration between pixel-space and textual reasoning. Targeting visual reasoning benchmarks (V* bench, TallyQA-Complex, InfographicsVQA, MVBench), it provides pre-trained models and evaluation scripts. The architecture combines adapted OpenRLHF components with custom VLM modules, Ray-based distributed training, and specialized data processors for multimodal input. Key entry points include training scripts (single/multi-node) and evaluation scripts, with data flowing from Hugging Face datasets through tokenization/image processing to PPO optimization loops.
Visual Spatial Tuning (VST) is a research framework for enhancing Vision-Language Models (VLMs) with human-like visuospatial abilities, from perception to reasoning. It implements a two-stage pipeline: supervised fine-tuning (SFT) on 4.1M spatial perception samples (VST-P) followed by reinforcement learning (RL) on 135K reasoning samples (VST-R). The framework extends to Vision-Language-Action (VLA) models for robotic manipulation evaluated on the LIBERO benchmark. The repository contains custom model implementations (Qwen2.5VL, Qwen3VL) with sequence parallelism and FSDP support, data preprocessing scripts, evaluation benchmarks (VSIBench, 3D object detection on SUN RGB-D, VLM benchmarks), and a distributed RL training system (verl) with reward functions and advantage estimation. While primarily a research codebase, it includes production-quality training infrastructure and reusable data utilities. The code is actively maintained with recent commits, but has medium technical drift due to tight version pinning and reliance on an internal ByteDance framework (VeOmni) that is not publicly available, which may hinder full reproducibility.
This repository provides the serving infrastructure and fine-tuning examples for MedSigLIP, a medical vision-language model (SigLIP variant) that encodes medical images and text into a shared embedding space. The codebase is dominated by a production-grade serving system: a Triton Inference Server backend wrapped in a Gunicorn application, deployed on Google Cloud Vertex AI. It includes modular data accessors for DICOM (whole-slide and generic), GCS, HTTP images, and inline bytes, with modality-specific preprocessing (ICC profile correction, windowing, dynamic range rescaling). The serving layer handles batch prediction, authentication, structured logging, and health checks. While notebooks demonstrate fine-tuning and data-efficient classifier training using Hugging Face, the core of the repository is the serving stack, making it primarily an application with strong data-research hybridity. The code is well-structured with clear abstractions (abstract data accessors, model runner, predictor), thorough unit tests, and reusable patterns like three-tier ICC profile caching and thread-safe logging. However, it is tightly coupled to Google Cloud and medical imaging standards, and some complexity arises from forking and dependency management.
Open-AutoGLM (Phone Agent) is an AI-powered mobile automation framework that uses vision-language models to understand phone screens and execute user tasks through automated touch, swipe, and text input operations. The system works by repeatedly capturing device screenshots, sending them to a vision model (AutoGLM-Phone-9B or compatible API), parsing the model's decision into actionable commands, and executing those commands via platform-specific debugging tools (ADB for Android, HDC for HarmonyOS, WebDriverAgent for iOS). The architecture employs a device factory pattern to abstract platform differences behind a unified interface, an OpenAI-compatible model client with streaming and performance metrics, and a command-based action handler that supports tap, type, swipe, launch, back, home, long press, and takeover operations. The repository is structured as both a CLI application and a Python library, with comprehensive configuration for 50+ Android apps, 60+ HarmonyOS apps, and 100+ iOS apps, along with bilingual system prompts (Chinese/English), configurable timing parameters, and callback hooks for sensitive operation confirmation and manual intervention. Notable features include remote debugging support, verbose step-by-step execution with model thinking output, and integration with third-party model hosting services.
Monkey is a research codebase for multimodal large language models (MLLMs) focused on high-resolution image understanding and document-level text comprehension. It provides model definitions (Qwen-based Monkey/TextMonkey and InternVL-based Mini-Monkey), training scripts with DeepSpeed, Gradio demos, and an extensive evaluation harness covering dozens of VQA, captioning, and document understanding benchmarks. The repo also includes a data generation pipeline using BLIP2, GRiT, SAM, PaddleOCR, and ChatGPT to create detailed image captions and instruction-following data. Architecturally, it is a collection of Python scripts and modules rather than a deployable service, with heavy reuse of external libraries (transformers, flash-attn, detectron2) and research-grade code that exhibits duplication, hardcoded paths, and mixed model families. While functionally complete for its intended research purpose, the codebase lacks modularity and polish for production reuse, but contains several high-quality, self-contained components (e.g., token compression via PerceiverResampler, window attention, VQA evaluation metrics) that are extractable for other MLLM projects.