This repository is a comprehensive survey paper on the intersection of knowledge graphs and multi-modal learning. It organizes the literature into two main directions: KG-driven multi-modal learning (KG4MM) and multi-modal knowledge graphs (MM4KG). The survey covers various tasks such as visual question answering, image classification, multi-modal named entity recognition, and knowledge graph completion, among others. Each section includes a list of relevant papers with links to their sources (e.g., arXiv, conference proceedings). The repository is maintained through updates that include adding new papers and fixing links, as evidenced by the recent commit in 2026 and the news items up to 2024-12.
Moondream is a lightweight, open-source vision-language model (VLM) library and inference framework. The core of the repository is a PyTorch model (`MoondreamModel`) that comprises a vision encoder, a text decoder (based on a transformer with optional MoE layers), and a region model for spatial grounding. It supports image captioning, visual question answering, object detection, pointing, and gaze detection through a unified API. The library is designed for efficient inference on both GPU and CPU, with support for quantization (via torchao) and LoRA-based fine-tuning. The codebase is cleanly structured into modular components for vision, text, and region processing, each with clear configuration dataclasses. The repository also includes a HuggingFace integration (`HfMoondream`) for easy loading via the transformers library, and several ready-to-use demos (Gradio apps, CLI samples). Beyond the core library, the repo contains a rich set of evaluation scripts (ChartQA, POPE, DocVQA, COCO mAP, etc.) that benchmark the model on standard VLM tasks, and finetuning examples that demonstrate how to adapt the model to custom datasets using reinforcement learning or supervised fine-tuning. Additionally, there are recipe applications (promptable video redaction, content moderation, gaze detection) that showcase practical use cases built on top of Moondream. While the primary character is an application/library, the presence of extensive evaluation and finetuning code gives it a hybrid research-engineering flavor. The code is well-maintained, with active commits and CI, and uses up-to-date dependencies, resulting in low technical drift.
This repository is a curated 'awesome list' of research resources on multimodal learning in medical imaging. It contains no application code, tests, or data pipelines; its sole substantive file is a large README.md that catalogs papers, datasets, benchmarks, and code repositories. Topics span medical vision-language models, report generation, visual question answering, retrieval-augmented generation (RAG), and multimodal LLMs. The list is organized by task and includes links to arXiv PDFs, GitHub code, and dataset sources, with badges indicating publication venues.
This repository implements multiple state-of-the-art Visual Question Answering (VQA) models from 2018, including bottom-up top-down attention, Bilinear Attention Networks (BAN), inter-intra modality attention, learning to count, and conditioned graph structures. It is built on Cyanogenoid's vqa-counting framework and provides a clean, modular codebase for training and evaluating VQA models on the VQA v2 dataset. The code includes data preprocessing scripts (feature extraction, vocabulary building), a flexible training pipeline with various loss functions and learning rate schedules, and evaluation scripts that compute per-category accuracy. While the code is well-structured for research purposes and contains reusable components like the counting module and fusion layers, it relies on deprecated PyTorch APIs (e.g., Variable), hardcoded paths, and lacks unit tests or environment definitions, making reproducibility challenging. The repository is primarily a research artifact for comparing attention mechanisms and counting strategies in VQA.