TinyLLaVA Factory is a modular Python framework for building, training, evaluating, and serving small-scale Large Multimodal Models (LMMs). It provides a pluggable architecture with separate components for vision towers (CLIP, SigLIP, DINOv2), language models (Phi, TinyLlama, Gemma, Qwen2, etc.), and connectors (MLP, QFormer, Resampler), all accessible via factory registries. The training pipeline supports two-stage pretraining (alignment) and finetuning with full, LoRA, or QLoRA recipes using DeepSpeed. It includes a comprehensive evaluation suite for standard benchmarks (VQA, GQA, SQA, POPE, MMMU, etc.) and a Gradio-based web demo. The codebase is well-organized and follows clean architectural patterns, though many modules are tightly coupled to the internal framework. It is actively maintained, with recent additions like a visualization tool and support for new models.
LongCat-Flash-Omni is an application providing inference capabilities for a 560B parameter omni-modal model that processes text, audio, image, and video inputs to generate text and/or audio outputs. The system features a modular architecture with separate modality encoders (audio, vision), a unified data processor for multimodal inputs, and a post-processor for converting model outputs to usable formats. The main entry point is longcat_omni_demo.py which orchestrates the inference pipeline, leveraging SGLang for efficient model serving. The repository demonstrates strong engineering practices in its modular design, particularly in the audio feature processing and image transformation components.
The repository implements LLaVA-VLA, a Vision-Language-Action model for robotic manipulation tasks. It extends the LLaVA framework to predict action sequences from visual inputs, proprioceptive state, and language instructions using a diffusion-based action head. The system includes training pipelines for fine-tuning on datasets like CALVIN, inference servers (Flask and Gradio) for real-time robot control, and evaluation tools for benchmarks. The architecture features modular vision encoders, projectors, and resamplers with factory patterns for flexibility, action tokenization for discrete action prediction, and integration with PEFT/DeepSpeed for efficient training.
Show-o and Show-o2 are research repositories for unified multimodal understanding and generation. They implement transformer-based models (1.3B, 1.5B, 7B parameters) that can perform text-to-image generation, image/video captioning, visual question answering, inpainting/extrapolation, and mixed-modality generation (interleaved text and images). The codebase is organized around two major model versions: Show-o uses discrete tokens with MAGVITv2 VAE and Phi-1.5 LLM, while Show-o2 moves to continuous latent space with WanVAE (3D causal VAE) and Qwen2.5 LLM. Training follows a multi-stage pipeline (pretraining, instruction tuning) with novel techniques like flow matching (via a modular `transport` package), dual-path vision encoders (SigLIP + patch embedding), and omni-attention masks that combine causal and bidirectional attention for multimodal sequences. The repository includes distributed training configs (DeepSpeed, multi-node), evaluation scripts for GenEval and DPG-Bench, and comprehensive YAML configuration files. This is primarily a data_research codebase: it produces research artifacts (models, papers at ICLR 2025 and NeurIPS 2025) rather than a production application. The code exhibits good modularity for a research project, with reusable components like the transport module, omni-attention masks, mixed dataloader, and attention masking utilities. However, it suffers from typical research-code issues: hardcoded internal infrastructure paths, reliance on non-public data pipelines (e.g., internal packages for RefinedWeb), and some code duplication between Show-o and Show-o2. Reproducibility is moderate—dependencies are pinned and configs are detailed, but many datasets are large-scale web sources that require manual setup, and the training scripts assume specific hardware (multi-node A100 clusters). The project is actively maintained (commits in early 2026) with a roadmap for future work, including mixed-modal generation and video support.