Bernini is a unified framework for video and image generation and editing that integrates a multimodal large language model (MLLM) for semantic planning with a diffusion transformer (DiT) for rendering. It supports tasks including text-to-image, image-to-image, text-to-video, video-to-video, reference-to-video, and reference-video-to-video. The architecture features modular pipelines for both full (with MLLM) and renderer-only modes, leveraging the VeOmni framework for distributed training and inference via Ulysses sequence parallelism. Key components include custom attention backend selection, flow matching scheduler, and utilities for weight loading and data processing.
OpenMontage is an open-source, agentic video production platform where an AI coding assistant (Claude Code, Cursor) orchestrates the entire pipeline. It provides a library of 100+ tools (video generation, audio, graphics, enhancement, analysis, avatar, subtitles) behind a scored provider selection layer, and 10+ production pipeline manifests (animated explainer, cinematic, documentary montage, talking head, etc.) that define stage order, approval gates, and tool assignments. The core architecture is a Python framework with a BaseTool class, ToolRegistry for auto-discovery, pipeline loader, checkpoint system, and a Remotion-based video composer. It also includes a Backlot living storyboard server for live production monitoring and an Ink Theater procedural puppetry engine. While primarily an application, it contains data/research elements such as ManimGL/ManimCE educational animation examples covering transformer internals, quantum mechanics, and ML concepts, as well as a CLIP-based corpus builder for documentary montage pipelines.
LongCat-Video is a production-grade video generation foundation model (13.6B parameters) that supports text-to-video, image-to-video, video continuation, and audio-driven avatar animation. The codebase implements a dense DiT-style transformer with 3D patch embedding, block-sparse attention (BSA) via custom Triton kernels, 3D rotary position embeddings (RoPE), and context parallelism for multi-GPU inference. It includes a coarse-to-fine generation pipeline (480p → 720p) using LoRA adapters for distillation and refinement, and an avatar variant that conditions on audio embeddings from Wav2Vec2 or Whisper. The architecture is monolithic, with entry points being demo scripts and a Streamlit UI. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, Triton, and optional OpenAI API for prompt enhancement. The code demonstrates advanced engineering practices such as INT8 quantization, KV-caching, and tiled VAE decoding, but suffers from some code duplication and tight coupling to specific model dimensions.
MAGREF is a video generation application that creates videos from text prompts and reference images using a diffusion model with subject disentanglement. The system implements a modular architecture separating model logic, configurations, distributed computing support, and utility functions. Key entry points include generate.py (main script) and shell scripts for single/multi-GPU inference. The primary data flow involves encoding text prompts with T5, processing reference images through CLIP and custom resizing/padding, running diffusion steps with a WanModel transformer, and decoding latents to video via VAE. External dependencies include PyTorch, Hugging Face Transformers and Diffusers, xfuser for context parallelism, and flash-attention for efficient computation.
This repository is a curated reference list of academic papers and projects focused on style transfer using diffusion models. It organizes resources into categories including image synthesis (text-driven, image-driven, LoRA-based methods), video synthesis, 3D generation, style datasets, and related surveys. The repository contains no executable code, data, or training scripts; it serves solely as a bibliographic guide for researchers. Content is presented in markdown format with tables linking to arXiv papers and GitHub repositories, supplemented by badges for stars and publication dates.
This repository is the official implementation of the SANA family of diffusion models for high-resolution image and video generation, developed by NVIDIA Research. It is primarily a research codebase, containing training and inference pipelines for multiple model variants: SANA (text-to-image), SANA-1.5, SANA-Sprint (few-step distillation), SANA-Video, SANA-Video 2.0, SANA-Streaming (long video), LongSANA, and SANA-WM (world model for camera-controlled video). The codebase also includes post-training with reinforcement learning (Sol-RL), ControlNet support, and various schedulers (flow matching, DPM-Solver, LCM, self-forcing). While the core purpose is research, the repository also provides deployable Gradio applications, conversion scripts to Hugging Face diffusers format, and integration with ComfyUI and SGLang, giving it a hybrid application character. The code is heavily optimized with custom Triton kernels for efficient attention (LiteMLA, Gated Delta Net) and includes distributed training utilities (FSDP, context parallelism). The repository is actively maintained with frequent commits and supports a wide range of model sizes (600M to 14B parameters).
Helios is a real-time long video generation system that implements text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation using a 3D transformer architecture. The system features a modular design with specialized optimizations for performance including Ahead-of-Time Compilation (AOTI), group offloading for VRAM reduction, context parallelism (Ulysses/Ring attention), and Flash Attention kernel optimizations. It employs a multi-stage training pipeline (Base → Mid → Distilled) to balance quality and sampling speed. The repository includes a Gradio web interface (app.py) for interactive generation, command-line inference scripts, training utilities, and a comprehensive evaluation suite measuring video quality across aesthetic, motion, semantic, and naturalness dimensions. While containing research elements for ablation studies and benchmarking, its primary function is as a deployable video generation application.
HunyuanVideo-1.5 is an open-source video generation model by Tencent, implemented as a Python library with CLI scripts for inference (`generate.py`) and training (`train.py`). It generates high-quality videos from text or image prompts using a diffusion transformer (DiT) architecture with 8.3B parameters, supporting resolutions up to 1080p via super-resolution. The codebase provides a modular pipeline (`HunyuanVideo_1_5_Pipeline`) that orchestrates prompt encoding (byT5 text encoder, SigLIP vision encoder), denoising with flow matching, and VAE decoding. It includes advanced features like SSTA (Selective and Sliding Tile Attention) for efficient long-video processing, step distillation, CFG distillation, and various caching mechanisms (DeepCache, TeaCache, TaylorCache). The repository also offers a training script with distributed support (FSDP, context parallelism), LoRA fine-tuning, and the Muon optimizer. While primarily an application for video generation, it contains research-grade model implementations and evaluation benchmarks. The code is well-structured with reusable neural network modules, but relies on many optional hardware-specific dependencies (flash-attn, flex-block-attn, SageAttention) and external services for prompt rewriting, which may introduce operational complexity. The license is a custom Tencent Hunyuan Community License with restrictions.
The repository provides a comprehensive framework for generative AI models, particularly focused on image and video generation. It integrates various diffusion-based architectures (Stable Diffusion, Stable Diffusion XL, Hunyuan DiT, Wan, etc.) and offers utilities for model loading, prompt engineering, and post-processing. The system supports text-to-image, text-to-video, image-to-video, and video-to-video generation, along with editing capabilities via ControlNet and LoRA. The architecture is modular, with core components for model management, prompting, and VRAM optimization, though it exhibits some duplication across model-specific pipelines.