This repository contains the research code for Video-SALMONN-2, a family of audio-visual large language models for video understanding, developed by Tsinghua University and ByteDance. It includes three generations: Video-SALMONN-2 (based on LLaVA), Video-SALMONN-2+ (based on Qwen2.5-VL), and Video-SALMONN-2 Pro (based on Qwen3-VL). The core contribution is a caption-enhanced training paradigm that tightly aligns audio and visual signals. The codebase provides full training pipelines for supervised fine-tuning (SFT) and direct preference optimization (DPO), along with evaluation scripts, a demo mode, and a vLLM-based serving plugin for deployment. While the project is predominantly research-focused, it includes application-like components such as CLI inference and OpenAI-compatible serving. The code exhibits high duplication across the three model variants, hardcoded paths to internal cluster storage, and brittle dependency pinning, making it challenging to reproduce outside the original environment. However, several components—such as the multimodal batch sampler, audio-video processing utilities, and QFormer-based fusion—are well-structured and potentially reusable for other multimodal research projects.
The repository 'ml-road' is primarily a collection of educational resources for machine learning, deep learning, and agentic AI, including curated links to courses, books, and recent topics. It also contains two small demo applications: one for Chinese character recognition using a convolutional neural network with batch normalization (TensorFlow 1.x) and another for MNIST handwritten digit recognition using a softmax regression (TensorFlow 1.x). The applications are runnable as command-line tools but require specific data directory structures and are tightly coupled to global flags, limiting their reusability. The codebase shows signs of technical drift due to reliance on deprecated TensorFlow 1.x APIs.