$ show TinyLLaVA/TinyLLaVA_Factory
TinyLLaVA Factory is a modular Python framework for building, training, evaluating, and serving small-scale Large Multimodal Models (LMMs). It provides a pluggable architecture with separate components for vision towers (CLIP, SigLIP, DINOv2), language models (Phi, TinyLlama, Gemma, Qwen2, etc.), and connectors (MLP, QFormer, Resampler), all accessible via factory registries. The training pipeline supports two-stage pretraining (alignment) and finetuning with full, LoRA, or QLoRA recipes using DeepSpeed. It includes a comprehensive evaluation suite for standard benchmarks (VQA, GQA, SQA, POPE, MMMU, etc.) and a Gradio-based web demo. The codebase is well-organized and follows clean architectural patterns, though many modules are tightly coupled to the internal framework. It is actively maintained, with recent additions like a visualization tool and support for new models.
The LLM identified 2 files worth highlighting for reuse:
The connector module provides an abstract base class `Connector` and several concrete implementations (LinearConnector, MLPConnector, QFormerConnector, ResamplerConnector, MoFMLPConnector) registered via a central factory. This pattern allows adding new connector types with minimal code changes. The base class handles weight loading and freezing, while each subclass implements the projector network. This design is clean, extensible, and reusable for other multimodal projects that need a pluggable vision-language connector.
The `Template` dataclass and its subclasses (PhiTemplate, LlamaTemplate, GemmaTemplate, etc.) define a configurable chat template system for formatting multimodal conversations before tokenization. The `Formatter` abstract class (StringFormatter, EmptyFormatter) decouples template strings from logic. The `encode` method produces input_ids and labels with proper masking of instruction tokens. This pattern is isolated, self-contained, and could be reused in any project that requires structured chat formatting with image tokens and loss masking.