$ show adobe-research/custom-diffusion
The code answers the research question of how to customize large text-to-image diffusion models with minimal data and minimal parameter updates, and it compares against DreamBooth and Textual Inversion. It includes reusable components for few-shot personalization, delta checkpoint handling, lightweight model compression, multi-concept weight composition, and CLIP/DINO evaluation, although the codebase shows research-level roughness: duplicated CompVis/diffusers paths, hardcoded paths, and several known bugs in conversion and composition utilities.
The LLM identified 5 files worth highlighting for reuse:
Retrieves real regularization images from LAION using CLIP embeddings and a KNN search endpoint. This is a reusable data acquisition pattern for any diffusion fine-tuning method that needs prior-preservation images, and it can be adapted to other CLIP-indexed image corpora.
Implements a CustomDiffusion dataset and collate function with random scale augmentation and attention-mask generation. The mask-based loss pattern lets small or zoomed objects contribute to training loss only in the relevant region, a non-obvious augmentation strategy reusable in other object-centric diffusion fine-tuning pipelines.
Compresses custom diffusion delta weights using SVD low-rank factorization, reducing per-concept storage to roughly 15 MB. The u/diag(s)/v factorization and reconstruction pattern is a reusable approach for compressing fine-tuned weight updates in other diffusion or transformer fine-tuning workflows.
Computes CLIP text-alignment, CLIP image-alignment, and DINO feature-alignment scores for generated images against reference concept images. This evaluation pattern is dataset-agnostic and can be reused as a standard metric harness for comparing text-to-image personalization methods.
Merges multiple fine-tuned concept delta weights via a closed-form optimization-based composition method. This allows combining independently trained concepts without joint training, and the linear-algebra pattern is reusable for other modular weight-editing or model-composition tasks, though the implementation has minor hardcoded token-id issues.