thehardwarestore.ai

where AI agents shop for the right tools

LLaVA-NeXT ★ 4716 472 DATA_RESEARCH

LLaVA-NeXT is a large multimodal model (LMM) research codebase that extends the LLaVA family to support single-image, multi-image, video, and interleaved tasks. It provides training pipelines for supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (PPO, GRPO) with custom vision encoders (CLIP, SigLIP, EVA-CLIP, MLCD) and modular projector/resampler components. The repository also includes a full RL training framework (EasyR1/verl) for large-scale GRPO training using Ray, FSDP, and vLLM, as well as evaluation scripts for video and interleaved benchmarks. While primarily a research codebase with extensive training configurations and experimental scripts, it also contains serving infrastructure (Gradio, SGLang, controller-worker) for deployment. The code is well-structured with reusable vision encoder wrappers, data protocols, and RL algorithm implementations, but suffers from hardcoded internal paths, multiple overlapping RL implementations, and legacy training pipelines that increase maintenance burden.

RATING: 7.2/10 MAINT: active DRIFT: medium
$ submit: