The Hugging Face Diffusers library is a comprehensive, production-grade Python library for diffusion model inference and fine-tuning. It provides a vast collection of pre-built pipelines (text-to-image, image-to-image, inpainting, video generation, audio generation, and more), model architectures (UNet, Transformers, Autoencoders), schedulers (DDPM, DDIM, Flow Matching, etc.), and a rich ecosystem of loaders (LoRA, IP-Adapter, textual inversion, single-file), optimizers (attention backends, caching, offloading, tensor parallelism), and guiders (CFG, PAG, MAG, TCFG). The architecture is highly modular: a central `DiffusionPipeline` base class with mixins for loading and configuration, alongside a newer `ModularPipeline` framework that composes generation from reusable, declarative blocks (encoders, denoisers, decoders). The codebase is actively maintained with extensive test coverage (over 200k lines of tests), CI/CD pipelines, and support for quantization (bitsandbytes, GGUF, torchao). It is the de facto standard library for state-of-the-art diffusion models in PyTorch.
SGLang is a high-performance inference serving system for large language models (LLMs) and multimodal models (vision, video, audio, diffusion). It provides an OpenAI-compatible API, a Python frontend DSL for programmable inference, and supports a wide range of hardware backends (NVIDIA CUDA, AMD ROCm, Intel XPU, Ascend NPU, Apple MLX). The architecture is a distributed serving runtime (SRT) with a scheduler, model executor, custom CUDA/Triton kernels, and advanced features like speculative decoding (EAGLE, MTP, NGRAM), disaggregated prefill/decode, hierarchical KV cache (HiCache), and quantization (FP8, NVFP4, AWQ, GPTQ). It also includes a Rust-based model gateway for routing and a multimodal generation pipeline for diffusion models. The system is designed for production-scale deployment, powering trillions of tokens daily across hundreds of thousands of GPUs.
Bernini is a unified framework for video and image generation and editing that integrates a multimodal large language model (MLLM) for semantic planning with a diffusion transformer (DiT) for rendering. It supports tasks including text-to-image, image-to-image, text-to-video, video-to-video, reference-to-video, and reference-video-to-video. The architecture features modular pipelines for both full (with MLLM) and renderer-only modes, leveraging the VeOmni framework for distributed training and inference via Ulysses sequence parallelism. Key components include custom attention backend selection, flow matching scheduler, and utilities for weight loading and data processing.
VibeVoice is a production-grade speech AI framework from Microsoft Research, providing both text-to-speech (TTS) and automatic speech recognition (ASR) capabilities. The repository delivers inference and deployment infrastructure for frontier speech models built on a Qwen2.5 LLM backbone with continuous speech tokenizers at 7.5 Hz and a next-token diffusion framework. It includes a vLLM plugin for high-performance serving with OpenAI-compatible APIs, FastAPI/WebSocket streaming ASR, real-time TTS demos, and Gradio-based interactive applications. The codebase is primarily an application deployment toolkit, with embedded research code for model architectures, tokenizers, and diffusion schedulers. It supports long-form multi-speaker TTS (up to 90 minutes), streaming TTS, and single-pass ASR with diarization across multiple languages. The repository also provides LoRA fine-tuning scripts for ASR adaptation and comprehensive demo scripts for both local and cloud deployment.
This repository is the official PyTorch library for Flow Matching generative models, developed by Meta FAIR. It provides a modular and extensible framework for implementing continuous, discrete, and Riemannian flow matching algorithms. The core library consists of four main components: probability paths (defining how data transforms from noise to samples), schedulers (noise/time schedules), solvers (ODE or CTMC integration for sampling), and loss functions (e.g., mixture path generalized KL divergence). It also includes utilities for common operations like model wrapping, manifold geometry (Euclidean, sphere, flat torus), and categorical sampling. The library is designed to be flexible, allowing researchers to easily combine different paths, schedulers, and solvers. Beyond the core library, the repository contains extensive examples for image generation (pixel-space continuous and discrete flow matching on CIFAR-10 and ImageNet) and text generation (discrete flow matching on FineWeb-Edu and Wikitext-103). These examples include full training pipelines, evaluation scripts, and model architectures (UNet, DiT-like transformer). The code is research-grade but well-structured, with clear separation of concerns and thorough unit tests for core components. While the examples are somewhat tied to specific datasets and hardware configurations, the library itself is a reusable foundation for any flow matching project. The repository is a hybrid of a research codebase and a production-ready library. The core modules are abstracted and documented, making them suitable for integration into other projects. However, the examples contain research artifacts like hardcoded paths and experimental features (torch.compile, mixed precision) that may require adaptation. Overall, this is a high-quality, actively maintained resource for anyone working on generative modeling with flow matching.
Pollinations is a large monorepo hosting a multi-modal generative AI platform. It provides a unified API gateway (gen.pollinations.ai) for text, image, video, audio, 3D, and embeddings generation, routing requests to dozens of providers (OpenAI, Anthropic, Google, Fal, Replicate, ElevenLabs, etc.) with fallback chains, caching, rate limiting, and billing. A separate enterprise service (enter.pollinations.ai) handles authentication (GitHub OAuth, API keys, device code flow), user dashboards, API key management, Stripe/Pollen billing, community-hosted model endpoints, and a quest/reward gamification system. The platform also includes a design system UI library (@pollinations/ui), a Node.js SDK, a CLI tool (polli-cli), an MCP server, and over 20 community apps (chat, AI dungeon master, image reimagine, Discord bots, etc.). Infrastructure includes Cloudflare Workers, Durable Objects, D1, R2, Tinybird for analytics, and GPU inference servers (Flux, Dreamshaper, Z-Image) deployed on Vast/IO.net. The architecture is service-oriented within a monorepo: each major service (gen, enter, media, pollinations.ai) is a separate Cloudflare Worker with its own wrangler.toml and dependencies. Shared packages (shared/, packages/) provide cross-cutting concerns like billing logic, model registry, auth middleware, HTTP error handling, and UI components. Data flows from user requests through middleware (auth, balance, cache, rate-limit, safety, model resolution) to provider-specific adapters, with usage events published to Tinybird for analytics and billing. The system is heavily tested with Vitest, including integration tests with Cloudflare Workers test environment and VCR snapshots. External dependencies include Cloudflare (Workers, D1, R2, KV, Durable Objects, Vectorize), Tinybird (analytics), Stripe (payments), GitHub (OAuth, quests), and numerous AI model providers. The codebase is predominantly TypeScript with some Python (Floret agent, GPU servers, social automation) and Rust (polli-core for Discord bot). While the core platform is well-maintained, there is significant drift in community apps and infrastructure scripts, and the sheer number of provider integrations creates maintenance surface area.
ACE-Step is a production-grade music generation system that implements a flow-matching diffusion model (ACEStepTransformer2D) with a custom DCAE autoencoder and vocoder. It provides multiple deployment options: a Gradio UI, CLI inference, and FastAPI REST endpoints. The system supports text-to-music, lyrics-conditioned generation, audio-to-audio editing, LoRA fine-tuning, and multilingual lyrics (17+ languages). Architecturally, it extends Hugging Face diffusers with custom attention processors (LiteLA linear attention), schedulers (Euler/Heun/PingPong flow matching), and a Conformer-based lyric encoder. It includes CPU offloading, overlapped decode, and memory optimizations for GPU-constrained environments. While primarily an application, the repository also contains training scripts (PyTorch Lightning) and dataset conversion utilities, reflecting its hybrid research-production nature.
GLM-Image is an image generation model implementing a hybrid autoregressive + diffusion decoder architecture. The autoregressive component (9B parameters) generates discrete image tokens from text, while the diffusion decoder (7B parameters) converts these tokens to pixel images. The repository provides tools for LoRA finetuning of the autoregressive component, image generation via API or direct model usage, prompt optimization utilities, and model comparison/visualization tools. Entry points include finetune_lora.py for training, glm_image_cli.py for API-based generation, and compare.py/plot_loss.py for evaluation and monitoring.
Audiocraft is a research codebase from Meta (Facebook Research) for state-of-the-art audio generation, including MusicGen, AudioGen, MAGNeT, JASCO, MultiBand Diffusion, and AudioSeal watermarking. It provides training and inference pipelines for text-to-music, text-to-sound, music continuation, style transfer, and symbolic conditioning (chords, drums, melody). The repository is primarily a research library with extensive configuration-driven training (Hydra/Dora), evaluation metrics (FAD, KLD, CLAP, ViSQOL), and interactive Gradio demos. It is not a production application but a collection of models, solvers, and utilities for audio generation research. The codebase is well-structured with modular components (conditioners, codebook patterns, audio effects) and includes unit tests, but relies on internal datasets and cluster paths that limit full reproducibility. Pretrained models are available via HuggingFace Hub for inference.
This repository is the official implementation of the SANA family of diffusion models for high-resolution image and video generation, developed by NVIDIA Research. It is primarily a research codebase, containing training and inference pipelines for multiple model variants: SANA (text-to-image), SANA-1.5, SANA-Sprint (few-step distillation), SANA-Video, SANA-Video 2.0, SANA-Streaming (long video), LongSANA, and SANA-WM (world model for camera-controlled video). The codebase also includes post-training with reinforcement learning (Sol-RL), ControlNet support, and various schedulers (flow matching, DPM-Solver, LCM, self-forcing). While the core purpose is research, the repository also provides deployable Gradio applications, conversion scripts to Hugging Face diffusers format, and integration with ComfyUI and SGLang, giving it a hybrid application character. The code is heavily optimized with custom Triton kernels for efficient attention (LiteMLA, Gated Delta Net) and includes distributed training utilities (FSDP, context parallelism). The repository is actively maintained with frequent commits and supports a wide range of model sizes (600M to 14B parameters).
Helios is a real-time long video generation system that implements text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation using a 3D transformer architecture. The system features a modular design with specialized optimizations for performance including Ahead-of-Time Compilation (AOTI), group offloading for VRAM reduction, context parallelism (Ulysses/Ring attention), and Flash Attention kernel optimizations. It employs a multi-stage training pipeline (Base → Mid → Distilled) to balance quality and sampling speed. The repository includes a Gradio web interface (app.py) for interactive generation, command-line inference scripts, training utilities, and a comprehensive evaluation suite measuring video quality across aesthetic, motion, semantic, and naturalness dimensions. While containing research elements for ablation studies and benchmarking, its primary function is as a deployable video generation application.
SimScale is a research codebase for data-driven, non-reactive autonomous vehicle simulation and benchmarking, developed by OpenDriveLab. It provides a comprehensive pipeline for training and evaluating end-to-end driving policies on the nuPlan/OpenScene dataset, with support for synthetic data augmentation (SimScale) and multiple agent architectures including Transfuser, DiffusionDrive, and GTRS-Dense. The repository implements a Plan-Do-Measure (PDM) closed-loop scoring framework, metric caching, and a Hydra-based configuration system for flexible experiment management. While primarily a research platform, it includes production-like training scripts, evaluation pipelines, and submission tools for the NAVSIM v2 challenge. The codebase is heavily coupled to the nuPlan ecosystem and requires large-scale data downloads (>1TB), but offers reusable components such as abstract agent interfaces, feature/target builders, and batch IDM policies. It is actively maintained with recent commits and a growing community of extensions.
This repository provides the official PyTorch implementation of TaDiCodec, a text-aware diffusion speech tokenizer accepted at NeurIPS 2025. The core contribution is a novel speech codec that achieves ultra-low bitrate (0.0875 kbps at 6.25 Hz) by combining a non-autoregressive Llama encoder, binary spherical quantization (BSQ), and a flow-matching diffusion decoder conditioned on text and prompt speech. The repo also includes token-based zero-shot TTS models: an autoregressive (AR) model using LLMs (Qwen2.5, Phi-3.5) and a masked diffusion (MGM) model. Inference pipelines with automatic model downloading from Hugging Face are provided. Training scripts for the tokenizer are partially implemented (dataloader, trainer class), but TTS model training scripts are incomplete. The project has migrated to a new repository (AmphionTeam/Diffusion-Speech-Tokenizer), and this repo is no longer the primary development branch, though recent commits indicate some ongoing activity.
This repository is an educational collection of minimal, from-scratch implementations of core LLM and generative AI systems, maintained by DataWhale. It is not a single production application; rather, it is a set of self-contained 'Tiny' sub-projects covering a small LLM trained on TinyStories, a RAG pipeline, a ReAct agent, an LLM evaluation harness, a DDPM diffusion model, a Transformer, a GraphRAG system, and an image-based RAG pipeline. Each sub-project has its own README, requirements, and runnable code or notebooks, with heavy Chinese-language comments explaining the underlying theory. The code is intentionally simple and readable rather than production-hardened, and it serves as a teaching resource for understanding how these systems work internally.