Monkey is a research codebase for multimodal large language models (MLLMs) focused on high-resolution image understanding and document-level text comprehension. It provides model definitions (Qwen-based Monkey/TextMonkey and InternVL-based Mini-Monkey), training scripts with DeepSpeed, Gradio demos, and an extensive evaluation harness covering dozens of VQA, captioning, and document understanding benchmarks. The repo also includes a data generation pipeline using BLIP2, GRiT, SAM, PaddleOCR, and ChatGPT to create detailed image captions and instruction-following data. Architecturally, it is a collection of Python scripts and modules rather than a deployable service, with heavy reuse of external libraries (transformers, flash-attn, detectron2) and research-grade code that exhibits duplication, hardcoded paths, and mixed model families. While functionally complete for its intended research purpose, the codebase lacks modularity and polish for production reuse, but contains several high-quality, self-contained components (e.g., token compression via PerceiverResampler, window attention, VQA evaluation metrics) that are extractable for other MLLM projects.
This repository implements multiple state-of-the-art Visual Question Answering (VQA) models from 2018, including bottom-up top-down attention, Bilinear Attention Networks (BAN), inter-intra modality attention, learning to count, and conditioned graph structures. It is built on Cyanogenoid's vqa-counting framework and provides a clean, modular codebase for training and evaluating VQA models on the VQA v2 dataset. The code includes data preprocessing scripts (feature extraction, vocabulary building), a flexible training pipeline with various loss functions and learning rate schedules, and evaluation scripts that compute per-category accuracy. While the code is well-structured for research purposes and contains reusable components like the counting module and fusion layers, it relies on deprecated PyTorch APIs (e.g., Variable), hardcoded paths, and lacks unit tests or environment definitions, making reproducibility challenging. The repository is primarily a research artifact for comparing attention mechanisms and counting strategies in VQA.