$ show LLaVA-VL/LLaVA-NeXT
LLaVA-NeXT is a large multimodal model (LMM) research codebase that extends the LLaVA family to support single-image, multi-image, video, and interleaved tasks. It provides training pipelines for supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (PPO, GRPO) with custom vision encoders (CLIP, SigLIP, EVA-CLIP, MLCD) and modular projector/resampler components. The repository also includes a full RL training framework (EasyR1/verl) for large-scale GRPO training using Ray, FSDP, and vLLM, as well as evaluation scripts for video and interleaved benchmarks. While primarily a research codebase with extensive training configurations and experimental scripts, it also contains serving infrastructure (Gradio, SGLang, controller-worker) for deployment. The code is well-structured with reusable vision encoder wrappers, data protocols, and RL algorithm implementations, but suffers from hardcoded internal paths, multiple overlapping RL implementations, and legacy training pipelines that increase maintenance burden.
The LLM identified 4 files worth highlighting for reuse:
Implements the EVA-CLIP vision transformer with custom attention mechanisms (xformers, scaled cosine attention), rotary position embeddings, and layer scaling. This is a self-contained, modular vision encoder that can be extracted and used in any vision-language model. It demonstrates non-obvious skill in adapting CLIP-style architectures for high-resolution image processing.
Provides core RL algorithm implementations including Generalized Advantage Estimation (GAE), policy/value loss functions, and KL controllers (adaptive and fixed). These are isolated, well-documented functions that can be reused in any RLHF training pipeline. The code demonstrates deep understanding of PPO/GRPO optimization and is not tied to a specific model architecture.
Defines DataProto, a flexible data transfer protocol for handling batched tensors, non-tensor data, and metadata with serialization, chunking, and concatenation. This is a reusable utility for distributed training systems that need to exchange heterogeneous data between workers. It is self-contained and demonstrates clean design for efficient data communication.
Contains image preprocessing utilities for high-resolution multimodal models, including anyres (any resolution) processing, padding, patch extraction, and dynamic image splitting. These functions are parameterized and can be reused in any vision-language pipeline that needs to handle variable-size images. The code shows careful engineering for efficient batched processing.