$ show bytedance/Bernini
Bernini is a unified framework for video and image generation and editing that integrates a multimodal large language model (MLLM) for semantic planning with a diffusion transformer (DiT) for rendering. It supports tasks including text-to-image, image-to-image, text-to-video, video-to-video, reference-to-video, and reference-video-to-video. The architecture features modular pipelines for both full (with MLLM) and renderer-only modes, leveraging the VeOmni framework for distributed training and inference via Ulysses sequence parallelism. Key components include custom attention backend selection, flow matching scheduler, and utilities for weight loading and data processing.
The LLM identified 4 files worth highlighting for reuse:
Provides a robust mechanism for selecting attention backends (FlashAttention-3, FlashAttention-2, or PyTorch SDPA) based on availability, with graceful fallbacks. This isolated utility solves a common problem in deep learning projects of handling optional GPU-accelerated dependencies without coupling the rest of the code to specific implementations.
Handles loading model weights from safetensors files, including support for exponential moving average (EMA) weights and handling of missing/unexpected keys. This self-contained module encapsulates best practices for weight loading in PyTorch projects and can be reused in other model repositories with minimal modification.
Contains utilities for loading and preprocessing images and videos, including resizing, center-cropping, and frame extraction. These functions are isolated from model-specific logic and provide robust, reusable handling of common media input tasks in computer vision workflows.
Implements a custom flow matching noise scheduler for diffusion sampling, featuring shift parameters and sigma schedule calculations. This self-contained component isolates the noise scheduling logic, making it adaptable to other diffusion models without requiring changes to the core training or inference loops.