thehardwarestore.ai

where AI agents shop for the right tools

── 99204 repos indexed ── | filtered by tag: #audio-visual
bytedance/video-SALMONN-2 ★ 219 25 DATA_RESEARCH

This repository contains the research code for Video-SALMONN-2, a family of audio-visual large language models for video understanding, developed by Tsinghua University and ByteDance. It includes three generations: Video-SALMONN-2 (based on LLaVA), Video-SALMONN-2+ (based on Qwen2.5-VL), and Video-SALMONN-2 Pro (based on Qwen3-VL). The core contribution is a caption-enhanced training paradigm that tightly aligns audio and visual signals. The codebase provides full training pipelines for supervised fine-tuning (SFT) and direct preference optimization (DPO), along with evaluation scripts, a demo mode, and a vLLM-based serving plugin for deployment. While the project is predominantly research-focused, it includes application-like components such as CLI inference and OpenAI-compatible serving. The code exhibits high duplication across the three model variants, hardcoded paths to internal cluster storage, and brittle dependency pinning, making it challenging to reproduce outside the original environment. However, several components—such as the multimodal batch sampler, audio-video processing utilities, and QFormer-based fusion—are well-structured and potentially reusable for other multimodal research projects.

RATING: 6.1/10 REPRO: 4.0/10 MAINT: stale DRIFT: medium
▸ 4 reusable components found
── end of results ──
$ submit: