$ show Yuliang-Liu/Monkey
Monkey is a research codebase for multimodal large language models (MLLMs) focused on high-resolution image understanding and document-level text comprehension. It provides model definitions (Qwen-based Monkey/TextMonkey and InternVL-based Mini-Monkey), training scripts with DeepSpeed, Gradio demos, and an extensive evaluation harness covering dozens of VQA, captioning, and document understanding benchmarks. The repo also includes a data generation pipeline using BLIP2, GRiT, SAM, PaddleOCR, and ChatGPT to create detailed image captions and instruction-following data. Architecturally, it is a collection of Python scripts and modules rather than a deployable service, with heavy reuse of external libraries (transformers, flash-attn, detectron2) and research-grade code that exhibits duplication, hardcoded paths, and mixed model families. While functionally complete for its intended research purpose, the codebase lacks modularity and polish for production reuse, but contains several high-quality, self-contained components (e.g., token compression via PerceiverResampler, window attention, VQA evaluation metrics) that are extractable for other MLLM projects.
The LLM identified 4 files worth highlighting for reuse:
Implements a Perceiver-style resampler that compresses a variable number of visual tokens into a fixed set of latent queries using cross-attention. It is self-contained (imports only torch, einops) and can be extracted to any vision-language model that needs to reduce the sequence length of visual features. The design uses learned latent tokens and layer normalization, demonstrating a clean application of the perceiver architecture.
Provides modular implementations of WindowAttention and CrossWindowAttention, inspired by Swin Transformer. These modules support relative position biases, attention masking, and efficient window-based self- and cross-attention. The code is well-structured with clear parameterization and could be reused in any transformer-based vision model that requires local attention with cross-window interaction.
Wraps the flash-attention CUDA kernel (from the flash-attn library) with a PyTorch autograd function that handles unpadding and padding of sequences. This component is self-contained and essential for efficient attention in long-sequence MLLM inference. It includes fallback to standard attention for unsupported dtypes, making it safe and reusable across different transformer architectures.
Contains reusable VQA evaluation metrics: VQAEval (exact match, VQA accuracy with answer processor), TextVQAAccuracyEvaluator, STVQAAccuracyEvaluator, and STVQAANLSEvaluator. These classes are self-contained (imports only python standard libs and a few utils) and can be extracted to evaluate any open-ended VQA model. The answer normalization logic (lowercase, punctuation removal, article stripping) is comprehensive and demonstrates strong understanding of VQA evaluation nuances.