This repository provides a comprehensive, production-grade training framework for fine-tuning the Qwen-VL family of vision-language models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5) using HuggingFace Transformers, Liger-Kernel, and DeepSpeed. It supports Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), and classification fine-tuning. The architecture is modular: dataset loaders handle mixed-modality (image/video/text) data with reasoning field support, custom trainer subclasses optimize memory and enable generation-based evaluation, and model wrappers provide sequence classification heads with configurable MLP bridges. Key engineering achievements include monkey-patched vision transformer forward passes for reduced memory usage, robust handling of DeepSpeed ZeRO-3 with dummy vision inputs for text-only batches, and a workaround for PEFT/DeepSpeed compatibility. The codebase is actively maintained, dependencies are pinned, and the project demonstrates strong craftsmanship with several reusable components.
LLaVA-NeXT is a large multimodal model (LMM) research codebase that extends the LLaVA family to support single-image, multi-image, video, and interleaved tasks. It provides training pipelines for supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (PPO, GRPO) with custom vision encoders (CLIP, SigLIP, EVA-CLIP, MLCD) and modular projector/resampler components. The repository also includes a full RL training framework (EasyR1/verl) for large-scale GRPO training using Ray, FSDP, and vLLM, as well as evaluation scripts for video and interleaved benchmarks. While primarily a research codebase with extensive training configurations and experimental scripts, it also contains serving infrastructure (Gradio, SGLang, controller-worker) for deployment. The code is well-structured with reusable vision encoder wrappers, data protocols, and RL algorithm implementations, but suffers from hardcoded internal paths, multiple overlapping RL implementations, and legacy training pipelines that increase maintenance burden.
This repository contains the research code for Video-SALMONN-2, a family of audio-visual large language models for video understanding, developed by Tsinghua University and ByteDance. It includes three generations: Video-SALMONN-2 (based on LLaVA), Video-SALMONN-2+ (based on Qwen2.5-VL), and Video-SALMONN-2 Pro (based on Qwen3-VL). The core contribution is a caption-enhanced training paradigm that tightly aligns audio and visual signals. The codebase provides full training pipelines for supervised fine-tuning (SFT) and direct preference optimization (DPO), along with evaluation scripts, a demo mode, and a vLLM-based serving plugin for deployment. While the project is predominantly research-focused, it includes application-like components such as CLI inference and OpenAI-compatible serving. The code exhibits high duplication across the three model variants, hardcoded paths to internal cluster storage, and brittle dependency pinning, making it challenging to reproduce outside the original environment. However, several components—such as the multimodal batch sampler, audio-video processing utilities, and QFormer-based fusion—are well-structured and potentially reusable for other multimodal research projects.
This repository is a comprehensive educational resource focused on large language models (LLMs), reinforcement learning (RL), and related training algorithms. It contains over 100 original diagrams (SVG/PNG/PDF) covering topics such as LLM architecture, SFT, DPO, RLHF, PPO, GRPO, policy gradient, and reasoning optimization. The repo also includes a curated index of open-source LLMs and VLMs with links to papers, code, and configs. While not a traditional software application or data pipeline, it serves as a visual knowledge base for researchers and practitioners in AI alignment and model training. The only executable code is a few utility scripts for image processing; the main value lies in the meticulously crafted diagrams that explain complex concepts with clarity. The repo is actively maintained and frequently updated with new content.