$ show bytedance/video-SALMONN-2
This repository contains the research code for Video-SALMONN-2, a family of audio-visual large language models for video understanding, developed by Tsinghua University and ByteDance. It includes three generations: Video-SALMONN-2 (based on LLaVA), Video-SALMONN-2+ (based on Qwen2.5-VL), and Video-SALMONN-2 Pro (based on Qwen3-VL). The core contribution is a caption-enhanced training paradigm that tightly aligns audio and visual signals. The codebase provides full training pipelines for supervised fine-tuning (SFT) and direct preference optimization (DPO), along with evaluation scripts, a demo mode, and a vLLM-based serving plugin for deployment. While the project is predominantly research-focused, it includes application-like components such as CLI inference and OpenAI-compatible serving. The code exhibits high duplication across the three model variants, hardcoded paths to internal cluster storage, and brittle dependency pinning, making it challenging to reproduce outside the original environment. However, several components—such as the multimodal batch sampler, audio-video processing utilities, and QFormer-based fusion—are well-structured and potentially reusable for other multimodal research projects.
The LLM identified 4 files worth highlighting for reuse:
Implements WeightedRoundRobinBatchSampler, a generic multimodal batch sampler that balances samples from different modalities (audio-video, video-only, audio-only, text-only) using weighted round-robin allocation. This is a reusable data engineering pattern for any multimodal training pipeline that needs controlled modality mixing.
Provides Qwen3VLMultiModalProcessor, which expands single placeholder tokens (e.g., <|video_pad|>) into per-frame interleaved vision and audio blocks. Supports both pre-tokenized and string prompts, and can be adapted to other multimodal models that require token-level modality interleaving.
Contains LazySupervisedDataset and DataCollatorForAVSupervisedDataset for loading video frames, audio spectrograms, and conversation data. Handles dynamic padding, spectrogram stacking, and DPO pair collation. The dataset pattern (lazy loading, modality-aware preprocessing) is a good reference for building scalable multimodal dataloaders.
Implements a Whisper-based audio encoder with window-level Q-Former and audio projection. The architecture splits audio into overlapping windows, applies the Whisper encoder, then compresses via Q-Former queries. This pattern is reusable for any audio feature extraction task requiring fixed-length representation from variable-length speech.