$ show 2U1/Qwen-VL-Series-Finetune
This repository provides a comprehensive, production-grade training framework for fine-tuning the Qwen-VL family of vision-language models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5) using HuggingFace Transformers, Liger-Kernel, and DeepSpeed. It supports Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), and classification fine-tuning. The architecture is modular: dataset loaders handle mixed-modality (image/video/text) data with reasoning field support, custom trainer subclasses optimize memory and enable generation-based evaluation, and model wrappers provide sequence classification heads with configurable MLP bridges. Key engineering achievements include monkey-patched vision transformer forward passes for reduced memory usage, robust handling of DeepSpeed ZeRO-3 with dummy vision inputs for text-only batches, and a workaround for PEFT/DeepSpeed compatibility. The codebase is actively maintained, dependencies are pinned, and the project demonstrates strong craftsmanship with several reusable components.
The LLM identified 5 files worth highlighting for reuse:
Replaces the default Qwen2.5-VL vision transformer forward pass with a patched version that implements windowed attention via `cu_window_seqlens` and `fullatt_block_indexes`. This reduces memory usage and improves training speed by partitioning spatial tokens into windows. The implementation handles dynamic grid sizes, padding, and gradient checkpointing. Extractable as a standalone optimization for any ViT that uses 2D-RoPE.
Provides mixed-modality forward passes for all Qwen-VL model types. The critical pattern is creating dummy pixel/tensor inputs when a batch has no images or videos, then adding a zero-scaled version of their embeddings to avoid DeepSpeed ZeRO-3 errors with conditional computation. This is a common pain point in multi-modal training, and the solution here is clean, portable, and supports both image and video placeholders.
A clean, standalone implementation of multi-class Focal Loss with optional class weights (alpha) and configurable gamma. Handles edge cases like empty alpha tensors and provides proper normalization to match CrossEntropyLoss scaling. Easily extractable for any classification task requiring focal loss with imbalanced classes.
Contains `collect_base_only_tensors`, a function that preserves weights present in the base checkpoint but absent from the PEFT-loaded model (e.g., Qwen3.5's multi-token-prediction head). It handles checkpoint conversion mappings, dtype alignment, and sanity checks to avoid duplicating weights. This is essential for producing fully functional merged checkpoints and is reusable for any PEFT workflow where the base model has extra modules.
Implements `patch_deepspeed_zero3_peft_hooks`, a workaround for a known incompatibility between DeepSpeed ZeRO-3 and PEFT's `ModulesToSaveWrapper` attribute delegation. By preventing PEFT wrappers from delegating private hook-state attributes, it fixes `ds_grads_remaining` errors. The function auto-detects whether the DeepSpeed version already contains the upstream fix, making it a safe, reusable patch for any project combining PEFT with DeepSpeed ZeRO-3.