TinyLLaVA Factory is a modular Python framework for building, training, evaluating, and serving small-scale Large Multimodal Models (LMMs). It provides a pluggable architecture with separate components for vision towers (CLIP, SigLIP, DINOv2), language models (Phi, TinyLlama, Gemma, Qwen2, etc.), and connectors (MLP, QFormer, Resampler), all accessible via factory registries. The training pipeline supports two-stage pretraining (alignment) and finetuning with full, LoRA, or QLoRA recipes using DeepSpeed. It includes a comprehensive evaluation suite for standard benchmarks (VQA, GQA, SQA, POPE, MMMU, etc.) and a Gradio-based web demo. The codebase is well-organized and follows clean architectural patterns, though many modules are tightly coupled to the internal framework. It is actively maintained, with recent additions like a visualization tool and support for new models.
This repository is a curated 'awesome list' of research resources on multimodal learning in medical imaging. It contains no application code, tests, or data pipelines; its sole substantive file is a large README.md that catalogs papers, datasets, benchmarks, and code repositories. Topics span medical vision-language models, report generation, visual question answering, retrieval-augmented generation (RAG), and multimodal LLMs. The list is organized by task and includes links to arXiv PDFs, GitHub code, and dataset sources, with badges indicating publication venues.