$ show NVlabs/DoRA
The repository provides the official PyTorch implementation of DoRA (Weight-Decomposed Low-Rank Adaptation), a parameter-efficient fine-tuning method. It integrates with Hugging Face PEFT and includes training/evaluation scripts for vision-language and language tasks (e.g., VQA, ScienceQA, commonsense reasoning). The architecture is modular, with core components like DoRA tuners, adapter controllers, and low-rank layers. Entry points include training scripts (finetune.py, train_dora.py), evaluation scripts (evaluate.py, run_llava.py), and serving tools (gradio_web_server.py). External dependencies include PyTorch, Transformers, PEFT, bitsandbytes, and dataset-specific libraries.
The LLM identified 4 files worth highlighting for reuse:
Implements the DoRA method by decomposing weights into magnitude and direction components. Self-contained PEFT tuner that can replace LoRA, demonstrating the paper's novel contribution with minimal coupling to other modules.
Manages multiple adapter modules for dynamic selection in multi-task learning. Reusable across architectures with clean separation of concerns, showing non-obvious skill in adapter state management and forwarding logic.
Handles saving/loading of PEFT adapter models and configurations. Decoupled from specific architectures, essential for model portability, and demonstrates robust utility for adapter persistence.
Merges LoRA/DoRA adapter weights into base model for deployment. Self-contained script with clear inputs/outputs, reusable for any PEFT adapter supporting merging, and practical for serving adapted models.