$ show merveenoyan/smol-vision
This repository is a curated collection of Jupyter notebooks and Python scripts that demonstrate fine-tuning, inference, quantization, and optimization techniques for state-of-the-art vision and multimodal models from the Hugging Face ecosystem. It covers a wide range of tasks including visual question answering, object detection, image classification, video understanding, multimodal retrieval, and preference alignment (DPO). The code is primarily educational, designed to be run interactively in Google Colab, with each notebook or script serving as a self-contained tutorial for a specific model or technique. Notable patterns include LoRA/QLoRA fine-tuning, knowledge distillation via a custom ImageDistilTrainer, any-to-any embedding for retrieval, and video frame downsampling for multimodal input. The repository is not a production library or application; it is a learning resource for engineers and researchers exploring vision-language models.
The LLM identified 6 files worth highlighting for reuse:
Contains a fully self-contained `ImageDistilTrainer` class that extends `transformers.Trainer` to perform knowledge distillation for image classification models. It implements a custom loss function combining student cross-entropy and KL divergence with teacher logits. The trainer is reusable across different image classification tasks and models, requiring only a teacher model, student model, and data collator. This demonstrates non-obvious skill in extending the Hugging Face Trainer API for distillation.
Provides a reusable `downsample_video()` function that extracts a fixed number of uniformly spaced frames from a video file using OpenCV, along with their timestamps. This function can be extracted and applied to any video understanding task that requires frame sampling, regardless of the specific model. It demonstrates a clean, parameterized approach to video preprocessing.
Implements an `encode_message()` function that produces normalized embeddings from text, image, video, or audio inputs using a multimodal embedding model (OmniEmbed). The function applies a chat template, processes each modality appropriately, and returns a single normalized embedding vector. This pattern is directly reusable for any multimodal retrieval or RAG pipeline, and demonstrates non-obvious skill in handling mixed modalities in a unified embedding path.
Defines a `DinoV3Linear` class that wraps a DINOv3 backbone with a trainable linear classification head. This class is a clean, reusable pattern for implementing linear probing on any vision transformer backbone. It handles grayscale-to-RGB conversion and can be extracted for use in other classification tasks without modification.
Contains a custom `DocVQADataset` class and `collate_fn` that demonstrate a reusable pattern for fine-tuning vision-language models on document QA tasks. The dataset appends a task-specific prefix (<DocVQA>) to prompts, and the collate function uses the processor to batch images and tokenize text. This pattern is transferable to any VLM fine-tuning task requiring custom dataset handling.
Demonstrates a reusable pattern for applying Direct Preference Optimization (DPO) to vision-language models. The notebook defines a chat template for the PaliGemma processor and a `process()` function that converts raw preference data (prompt, chosen, rejected, images) into formatted inputs suitable for `DPOTrainer` from TRL. This pattern can be extracted and adapted for other multimodal DPO tasks, showcasing non-obvious skill in combining TRL with vision models.