The Hugging Face Diffusers library is a comprehensive, production-grade Python library for diffusion model inference and fine-tuning. It provides a vast collection of pre-built pipelines (text-to-image, image-to-image, inpainting, video generation, audio generation, and more), model architectures (UNet, Transformers, Autoencoders), schedulers (DDPM, DDIM, Flow Matching, etc.), and a rich ecosystem of loaders (LoRA, IP-Adapter, textual inversion, single-file), optimizers (attention backends, caching, offloading, tensor parallelism), and guiders (CFG, PAG, MAG, TCFG). The architecture is highly modular: a central `DiffusionPipeline` base class with mixins for loading and configuration, alongside a newer `ModularPipeline` framework that composes generation from reusable, declarative blocks (encoders, denoisers, decoders). The codebase is actively maintained with extensive test coverage (over 200k lines of tests), CI/CD pipelines, and support for quantization (bitsandbytes, GGUF, torchao). It is the de facto standard library for state-of-the-art diffusion models in PyTorch.
ComfyUI is a modular, node-based AI engine for content creation, primarily focused on generative image, video, audio, and 3D model inference. It provides a web server (aiohttp) with WebSocket and REST APIs, a visual workflow editor, and an extensible plugin system for custom nodes. The architecture is a monolith with a clear separation of concerns: core model loading and sampling (comfy/), execution engine (comfy_execution/), asset management (app/assets/), external API integrations (comfy_api_nodes/), and a public API for defining nodes (comfy_api/). It supports a vast range of open-source diffusion models (Flux, SD3, Wan, Hunyuan, etc.) and offers advanced features like dynamic VRAM management, quantization, multi-GPU support, and experimental guidance methods. The system operates as a local-first application but also integrates with cloud services via partner API nodes. Key entry points include main.py (startup), server.py (PromptServer), and execution.py (PromptExecutor). The codebase demonstrates high craftsmanship in its weight adapter hierarchy (LoRA, LoHa, OFT), patcher extension system for model hooks, and asset management services with database-backed metadata extraction and cursor-based pagination. However, it also contains experimental and deprecated nodes, some code duplication across model implementations, and a reliance on external dependencies that may drift over time.
Irodori-TTS is a state-of-the-art Japanese text-to-speech system built on rectified flow diffusion over DACVAE audio latents. It provides a complete training and inference stack: `train.py` supports full fine-tuning, LoRA, speaker inversion, and MeanFlow distillation with distributed training and WSD scheduling; `infer.py` offers a rich CLI with CFG, Sway Sampling, duration scaling, and watermarking; `gradio_app.py` and `gradio_app_voicedesign.py` expose interactive UIs for reference-based cloning and caption-conditioned voice design. The architecture is modular (`irodori_tts` package) with attention dispatch (FA3/SDPA), a hybrid Muon+AdamW optimizer, codec abstraction, and caching-friendly `InferenceRuntime`. While highly engineered for production use (INT4/INT8 quantization, safetensors, HuggingFace Hub integration), it remains a research-oriented codebase with experimental sampling algorithms and many YAML configs for different model sizes/versions. The code is clean and well-factored, though `train.py` is a large monolith and external dependencies (DACVAE, silentcipher) are pinned to specific commits, creating ecosystem fragility.
RectifiedFlow is a unified PyTorch library for implementing Rectified Flow models, a type of generative model that learns ordinary differential equations to transform data between distributions. The library provides a modular framework for training various architectures (MLP, UNet, DiT, LightningDiT) on different datasets, with support for multiple sampling algorithms and interpolation schemes. Key features include a symbolic solver for affine interpolation equations, utilities for converting between different interpolation schemes while preserving rectified flow properties, and a flexible sampler base class for implementing custom integration methods. The codebase demonstrates strong software engineering practices with clear separation of components, comprehensive documentation through examples, and active maintenance.
This repository is a comprehensive collection of annotated PyTorch implementations of deep learning papers, spanning transformers, GANs, reinforcement learning, diffusion models, and more. Each implementation includes side-by-side notes explaining the code and underlying concepts, making it an educational resource for understanding and reproducing paper results. The codebase is organized by paper/model type, with standalone experiment scripts, reusable modules, and configuration-driven training loops. Key strengths include clean abstractions for transformer components, a flexible configs system, and extensive coverage of both foundational and modern architectures. However, the project has not seen commits since early 2024, some dependencies are outdated, and certain experiments require large GPU resources or external data not bundled in the repo. While individual components are well-crafted and reusable, the overall collection is best used as a reference and learning tool rather than a production framework.
This repository contains the public-facing source code for Draw Things, a production-grade AI image and video generation application for macOS/iOS/Linux. The codebase implements a comprehensive suite of diffusion models (Stable Diffusion, FLUX, HiDream, Qwen, MiniMax, etc.), samplers, LoRA training, model conversion, and a gRPC-based distributed inference server. It is written primarily in Swift, using Bazel for builds, and includes extensive use of FlatBuffers for data serialization, SQLite for persistence, and a custom neural network compiler (NNC) for on-device inference. The architecture is modular, with separate libraries for tokenization (CLIP, Tiktoken, SentencePiece), model zoos (downloadable model specifications), image generation pipelines, and cloud authentication. The repository also includes CLI tools (draw-things-cli, gRPCServerCLI, model converters) and server management scripts for deploying GPU-backed inference services with Tailscale, Envoy, and Docker. While the core application is a consumer-facing app, the codebase also contains research-grade components such as instruction count estimators for performance profiling, formal verification tools (Z3, Lean), and web search integration with browser fallback. The code is well-structured with many reusable patterns, though it carries some maintenance overhead from vendored dependencies and generated code.
This repository is a curated reference list of academic papers and projects focused on style transfer using diffusion models. It organizes resources into categories including image synthesis (text-driven, image-driven, LoRA-based methods), video synthesis, 3D generation, style datasets, and related surveys. The repository contains no executable code, data, or training scripts; it serves solely as a bibliographic guide for researchers. Content is presented in markdown format with tables linking to arXiv papers and GitHub repositories, supplemented by badges for stars and publication dates.
Open-Sora is a video generation framework implementing text-to-video, image-to-video, and video-to-video capabilities using diffusion models and variational autoencoders. The system architecture consists of modular components for models (MMDiT, DC-AE, Hunyuan VAE), datasets with bucket-based sampling, acceleration utilities for tensor/sequence parallelism, and training/inference scripts. Key entry points include training scripts in `scripts/diffusion/train/` and `scripts/vae/train.py`, inference scripts in `scripts/diffusion/inference.py` and `scripts/vae/inference.py`, and a Gradio demo at `gradio/app.py`. The system relies on PyTorch, ColossalAI for distributed training, xformers/flash-attention for efficient attention, and safetensors for model checkpointing. Data flows involve loading video datasets, applying spatiotemporal transforms, training diffusion models with classifier-free guidance, and generating videos through denoising loops.
This repository is a curated awesome list and survey companion for diffusion models applied to time series, spatio-temporal, and tabular data. It contains no executable code; the sole content is a large README.md that organizes hundreds of papers into a taxonomy by task (prediction, generation, imputation, anomaly detection, classification/regression, causal inference, event prediction, foundation models) and by application domain (healthcare, sequential recommendation, weather, energy, finance, AIOps, environment, traffic, manufacturing, audio, biology, communication). Each entry links to the paper and, where available, official code repositories. It is the companion to an ACM Computing Surveys paper (DOI 10.1145/3783986) and includes citation information, a Google Sheets taxonomy, and an overview figure. The repository is actively maintained with recent commits and welcomes pull requests. As a research resource, it is highly reproducible: all content is in the markdown file, with no dependencies, datasets, or environment setup required. Although classified as data_research due to its research orientation, it is not a data pipeline or ML codebase. Its value is as a reference index and literature map for researchers; the README itself is the reusable asset.
Open-OmniVCus is a research monorepo reproducing the NeurIPS 2025 OmniVCus method for subject-driven video customization under multimodal control conditions (reference image, depth, mask, motion/camera). It is hybrid by nature: the dominant character is an application/framework layer provided by a large vendored/forked copy of DiffSynth-Studio, wrapped around research-oriented data-construction and training code (VideoCus-Factory, Video-Depth-Anything). The repo does not expose a hosted service or API; it is a Python library plus executable training/inference scripts for diffusion-based image and video generation.
This repository is a modified/forked version of the Hugging Face Diffusers library with specific additions for RealRestorer, a generalizable real-world image restoration model. It provides pipelines for various image restoration tasks (blur removal, compression artifact removal, lens flare removal, moire removal, dehazing, low-light enhancement, denoising, rain removal, reflection removal), tools for synthesizing real-world degradations, benchmark evaluation capabilities, and standard diffusion model functionalities. The architecture follows a modular library structure with lazy loading patterns, standardized pipeline interfaces, and extensive training/inference utilities. Key entry points include CLI commands for export, inference, and degradation synthesis, along with server examples for deployment.
graph_weather is a research-oriented deep learning library for weather forecasting with graph neural networks. It implements multiple model families—GraphCast-inspired encoder/processor/decoder models, the diffusion-based GenCast and GenDA, the ensemble FGN, WeatherMesh, Aurora, FengWu-GHR, and CaFA—along with a substantial set of data loaders, loss functions, training scripts, and evaluation utilities. The package is structured as a Python library with HuggingFace Hub model integration, PyTorch Lightning training support, and a comprehensive test suite, giving it a hybrid character: it is fundamentally research code, but packaged and tested with application-grade discipline. Entry points include GraphWeatherForecaster, GraphWeatherAssimilator, WeatherStationReader, and various dataset classes, while training scripts live under train/.
WorldGen is a Python library and demo application for generating 3D scenes from text prompts or single images in seconds. It orchestrates a pipeline of state-of-the-art models: a fine-tuned FLUX diffusion model to generate 360° panoramic images, DA-2 for monocular depth estimation on equirectangular images, and optionally OneFormer for semantic segmentation and LaMa for inpainting to handle background occlusions. The 3D scene is produced as a Gaussian splatting point cloud (via a custom RGBD-to-Gaussian conversion) or as a textured mesh. An experimental mode uses Apple's Sharp model for higher-quality Gaussian splats by processing cubemap faces. The codebase demonstrates strong engineering practices for a research project: modular pipeline design, use of submodules for dependencies, custom diffusers pipelines with VAE tiling modifications for seamless panorama generation, and a Viser-based interactive viewer. While the core value is the integrated pipeline, several utility modules (equirectangular projection utilities, splat file handling) are cleanly isolated and reusable in other 360° or 3D vision projects.
Stable Diffusion Web UI is a Gradio-based web application that provides a comprehensive interface for generating and manipulating images using diffusion models. It supports txt2img, img2img, inpainting, outpainting, and various post-processing operations including face restoration and upscaling. The application implements a plugin architecture through extensions, supports multiple model architectures (SD 1.x, SD 2.x, SDXL, SD3), and provides a REST API for programmatic access. The codebase is a monolithic application with tightly coupled modules handling model loading, image processing, UI rendering, and script execution. The architecture centers around a main webui.py entry point that initializes the application, sets up Gradio interfaces, and manages the event loop. The modules directory contains core functionality including sd_models.py for model management, processing.py for the main generation pipeline, and various hijack modules that monkey-patch diffusion model components to add custom behavior like attention optimization and CLIP modification. The extensions-builtin directory contains first-party extensions for LoRA, hypernetworks, and various post-processing operations. Key technical patterns include extensive use of monkey-patching and function hijacking to modify model behavior without changing upstream code, a callback system for script hooks, and a queue-based generation system with progress tracking. The codebase demonstrates sophisticated handling of GPU memory management, model optimization techniques, and cross-platform compatibility. However, the code exhibits significant technical debt with large monolithic files, tight coupling between modules, and complex conditional logic handling multiple model versions and configurations. The repository includes comprehensive test coverage for API endpoints, model utilities, and image processing functions. Dependencies are well-pinned in requirements_versions.txt, though some versions are notably behind current releases (Gradio 3.41.2, Transformers 4.30.2). The codebase shows active maintenance with recent commits and a detailed changelog tracking features and bug fixes across multiple versions.
This repository is the official implementation of the SANA family of diffusion models for high-resolution image and video generation, developed by NVIDIA Research. It is primarily a research codebase, containing training and inference pipelines for multiple model variants: SANA (text-to-image), SANA-1.5, SANA-Sprint (few-step distillation), SANA-Video, SANA-Video 2.0, SANA-Streaming (long video), LongSANA, and SANA-WM (world model for camera-controlled video). The codebase also includes post-training with reinforcement learning (Sol-RL), ControlNet support, and various schedulers (flow matching, DPM-Solver, LCM, self-forcing). While the core purpose is research, the repository also provides deployable Gradio applications, conversion scripts to Hugging Face diffusers format, and integration with ComfyUI and SGLang, giving it a hybrid application character. The code is heavily optimized with custom Triton kernels for efficient attention (LiteMLA, Gated Delta Net) and includes distributed training utilities (FSDP, context parallelism). The repository is actively maintained with frequent commits and supports a wide range of model sizes (600M to 14B parameters).
Helios is a real-time long video generation system that implements text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation using a 3D transformer architecture. The system features a modular design with specialized optimizations for performance including Ahead-of-Time Compilation (AOTI), group offloading for VRAM reduction, context parallelism (Ulysses/Ring attention), and Flash Attention kernel optimizations. It employs a multi-stage training pipeline (Base → Mid → Distilled) to balance quality and sampling speed. The repository includes a Gradio web interface (app.py) for interactive generation, command-line inference scripts, training utilities, and a comprehensive evaluation suite measuring video quality across aesthetic, motion, semantic, and naturalness dimensions. While containing research elements for ablation studies and benchmarking, its primary function is as a deployable video generation application.
Amphion is a comprehensive open-source research toolkit for audio, music, and speech generation, developed by OpenMMLab. It covers multiple generative tasks including text-to-speech (TTS), singing voice conversion (SVC), voice conversion (VC), text-to-audio (TTA), neural audio codecs, and vocoders. The codebase is organized around config-driven experiments (JSON + shell scripts) with modular base classes for trainers, datasets, and inference. It ingests a wide variety of public datasets (LJSpeech, LibriTTS, VCTK, M4Singer, Emilia, etc.) and provides extensive preprocessing pipelines for feature extraction (mel-spectrograms, F0, energy, phonetic alignments, self-supervised features from HuBERT/Whisper). The primary research focus is on high-quality, zero-shot, and controllable generation using architectures like diffusion models, flow matching, GANs, autoregressive transformers, and vector-quantized codecs. The repository also includes evaluation metrics (F0, energy, intelligibility, speaker similarity) and visualization tools (SingVisio). While primarily a research codebase, it includes inference scripts and Gradio demos for practical use. External dependencies include PyTorch, HuggingFace Transformers, WeNet, Descript Audio Codec, and various pretrained models.
Musubi Tuner is a comprehensive LoRA training and inference toolkit for a wide range of diffusion model architectures, including HunyuanVideo, Wan, FramePack, FLUX variants, Qwen-Image, Z-Image, HiDream-O1, Ideogram4, Krea2, and Kandinsky5. It provides a unified pipeline of pre-caching (latents and text encoder outputs), training (LoRA, LoHa, LoKr), and generation, all accessible via CLI scripts and an optional Gradio GUI. The codebase is built on a shared `NetworkTrainer` base class with architecture-specific overrides, and includes advanced memory-saving techniques such as block-swap CPU offloading, FP8 quantization, and gradient checkpointing. While the repository is actively maintained and widely used (2035 stars), it exhibits medium technical drift due to vendored external model code, deprecated diffusers API usage, and a proliferation of near-duplicate architecture-specific scripts. The core reusable components—unified attention backend, offloading utilities, FP8 optimization, and LoRA network infrastructure—demonstrate solid engineering craftsmanship.
SimpleFold is a research repository focused on protein structure prediction using flow-matching generative models. While it includes application-like features such as a CLI and inference scripts, its core architecture is designed for ML research, specifically training and evaluating large-scale diffusion transformers (FoldingDiT) on structural biology data. The codebase is hybrid in nature, combining rigorous data engineering pipelines (parsing MMCIF, filtering, tokenization) with standard deep learning frameworks (PyTorch Lightning, Hydra). It supports dual backends (PyTorch and MLX) for inference, indicating a focus on accessibility across hardware platforms. The repository's primary value lies in its data processing pipeline, which is heavily adapted from the Boltz-1 project. It handles complex structural data (PDB, SwissProt, AFESM) through a modular pipeline including parsing, filtering, and tokenization. The model architecture implements flow-matching with ESM-2 embeddings for sequence context. Evaluation metrics include pLDDT confidence, TM-score, and RMSD, typical for structural biology research. However, the codebase exhibits signs of research-to-production transition challenges. Hardcoded paths (e.g., `/mnt/task_runtime/...`) and reliance on external binaries (TMscore, OpenStructure) reduce portability. The data pipeline is a fork of an external project (`boltz`), introducing maintenance overhead for divergence. Despite these issues, the use of Hydra for configuration management and PyTorch Lightning for training orchestration demonstrates solid engineering practices for an ML research project.
VideoCoF is a unified video editing framework based on diffusion transformers (Wan2.1). It implements a 'See → Reason → Edit' pipeline using chain-of-frames (CoT) reasoning, enabling object removal, addition, swap, and local style transfer. The architecture includes a modular pipeline (`WanPipeline`) with flow matching solvers, VAE, text encoder, and transformer backbone. Inference can be done via a fast inference script (`fast_infer.py`) using 4-step DMD, batch inference via JSON, or through a Gradio UI. Training supports LoRA fine-tuning with DeepSpeed ZeRO-2/3 for 1.3B and 14B models, with custom dataset classes (including CoT and joint image-video). External dependencies include Diffusers, Transformers, Gradio, OpenCV, DeepSpeed, and optional paifuser for optimizations. While primarily an application for video editing, the repo has strong research character with extensive evaluation metrics (CLIP, GPT), configuration files, and experiment scripts. The code is modular in parts but contains experimental features, hardcoded paths, and reliance on an internal optimization package (paifuser) that may limit external reuse. The core utility modules (LoRA, caching, attention, schedulers) are well-crafted and potentially reusable across other diffusion model projects.