Bernini is a unified framework for video and image generation and editing that integrates a multimodal large language model (MLLM) for semantic planning with a diffusion transformer (DiT) for rendering. It supports tasks including text-to-image, image-to-image, text-to-video, video-to-video, reference-to-video, and reference-video-to-video. The architecture features modular pipelines for both full (with MLLM) and renderer-only modes, leveraging the VeOmni framework for distributed training and inference via Ulysses sequence parallelism. Key components include custom attention backend selection, flow matching scheduler, and utilities for weight loading and data processing.
Chatterbox is a state-of-the-art open-source text-to-speech library by Resemble AI, providing multiple TTS models (Chatterbox, Chatterbox-Multilingual, Chatterbox-Turbo, Chatterbox-Nano) and voice conversion. The architecture is modular: a T3 transformer generates speech tokens from text conditioned on speaker embeddings and optional reference audio; an S3Gen pipeline converts tokens to mel-spectrograms via conditional flow matching (CFM) and then to waveforms via a HiFiGAN vocoder. Voice cloning is achieved through a separate voice encoder (LSTM-based speaker embedding) and S3 tokenizer for prompt conditioning. The library supports 23+ languages with sophisticated multilingual tokenization (Cangjie for Chinese, hiragana for Japanese, diacritics for Hebrew, stress for Russian, Jamo decomposition for Korean). All models include imperceptible neural watermarking via Perth. The codebase is well-structured, with clean separation of concerns, efficient inference (KV caching, CFG, distilled single-step decoder for Turbo/Nano), and pinned dependencies. It is primarily an application library but includes research-grade model implementations.
This repository is the official PyTorch library for Flow Matching generative models, developed by Meta FAIR. It provides a modular and extensible framework for implementing continuous, discrete, and Riemannian flow matching algorithms. The core library consists of four main components: probability paths (defining how data transforms from noise to samples), schedulers (noise/time schedules), solvers (ODE or CTMC integration for sampling), and loss functions (e.g., mixture path generalized KL divergence). It also includes utilities for common operations like model wrapping, manifold geometry (Euclidean, sphere, flat torus), and categorical sampling. The library is designed to be flexible, allowing researchers to easily combine different paths, schedulers, and solvers. Beyond the core library, the repository contains extensive examples for image generation (pixel-space continuous and discrete flow matching on CIFAR-10 and ImageNet) and text generation (discrete flow matching on FineWeb-Edu and Wikitext-103). These examples include full training pipelines, evaluation scripts, and model architectures (UNet, DiT-like transformer). The code is research-grade but well-structured, with clear separation of concerns and thorough unit tests for core components. While the examples are somewhat tied to specific datasets and hardware configurations, the library itself is a reusable foundation for any flow matching project. The repository is a hybrid of a research codebase and a production-ready library. The core modules are abstracted and documented, making them suitable for integration into other projects. However, the examples contain research artifacts like hardcoded paths and experimental features (torch.compile, mixed precision) that may require adaptation. Overall, this is a high-quality, actively maintained resource for anyone working on generative modeling with flow matching.
Irodori-TTS is a state-of-the-art Japanese text-to-speech system built on rectified flow diffusion over DACVAE audio latents. It provides a complete training and inference stack: `train.py` supports full fine-tuning, LoRA, speaker inversion, and MeanFlow distillation with distributed training and WSD scheduling; `infer.py` offers a rich CLI with CFG, Sway Sampling, duration scaling, and watermarking; `gradio_app.py` and `gradio_app_voicedesign.py` expose interactive UIs for reference-based cloning and caption-conditioned voice design. The architecture is modular (`irodori_tts` package) with attention dispatch (FA3/SDPA), a hybrid Muon+AdamW optimizer, codec abstraction, and caching-friendly `InferenceRuntime`. While highly engineered for production use (INT4/INT8 quantization, safetensors, HuggingFace Hub integration), it remains a research-oriented codebase with experimental sampling algorithms and many YAML configs for different model sizes/versions. The code is clean and well-factored, though `train.py` is a large monolith and external dependencies (DACVAE, silentcipher) are pinned to specific commits, creating ecosystem fragility.
ACE-Step is a production-grade music generation system that implements a flow-matching diffusion model (ACEStepTransformer2D) with a custom DCAE autoencoder and vocoder. It provides multiple deployment options: a Gradio UI, CLI inference, and FastAPI REST endpoints. The system supports text-to-music, lyrics-conditioned generation, audio-to-audio editing, LoRA fine-tuning, and multilingual lyrics (17+ languages). Architecturally, it extends Hugging Face diffusers with custom attention processors (LiteLA linear attention), schedulers (Euler/Heun/PingPong flow matching), and a Conformer-based lyric encoder. It includes CPU offloading, overlapped decode, and memory optimizations for GPU-constrained environments. While primarily an application, the repository also contains training scripts (PyTorch Lightning) and dataset conversion utilities, reflecting its hybrid research-production nature.
MAGREF is a video generation application that creates videos from text prompts and reference images using a diffusion model with subject disentanglement. The system implements a modular architecture separating model logic, configurations, distributed computing support, and utility functions. Key entry points include generate.py (main script) and shell scripts for single/multi-GPU inference. The primary data flow involves encoding text prompts with T5, processing reference images through CLIP and custom resizing/padding, running diffusion steps with a WanModel transformer, and decoding latents to video via VAE. External dependencies include PyTorch, Hugging Face Transformers and Diffusers, xfuser for context parallelism, and flash-attention for efficient computation.
MeanFlow is a PyTorch implementation of a novel single-step flow matching model for high-quality image generation. The codebase provides a complete pipeline for training and evaluating SiT (Scalable Interpolant Transformer) models on ImageNet 256x256 and CIFAR-10, with a specific focus on achieving competitive FID scores in a single sampling step. Key components include a custom loss function with bootstrap and classifier-free guidance, a flexible time-sampling strategy, and distributed training/evaluation scripts using Accelerate. The repository also contains preprocessing utilities to convert ImageNet into VAE-encoded latent LMDB databases. While primarily a research codebase, it demonstrates strong engineering practices with clean module separation, EMA tracking, and comprehensive argument configuration. The implementation closely reproduces the MeanFlow paper results but requires significant computational resources (multiple GPUs, large storage) to run.
TRELLIS.2 is a research codebase for state-of-the-art image-to-3D generation using a novel sparse voxel representation (O-Voxel) and flow matching with 4B parameters. It provides full training and inference pipelines for generating high-fidelity 3D meshes with PBR materials from single images. The repository includes custom CUDA extensions for voxel processing, a flexible sparse tensor library, and comprehensive dataset tooling for large-scale 3D asset curation (Objaverse-XL, Sketchfab). While primarily a research project with training code and data pipelines, it also offers Gradio web demos and Python APIs for inference. The codebase demonstrates strong engineering in its modular design (pipelines, mixins, config-driven training) but suffers from some research-typical issues like code duplication, hardcoded paths, and fragile dependency management.
This is a research-grade PyTorch implementation of Danijar Hafner's Dreamer v4 model-based reinforcement learning algorithm. The core artifact is the `dreamer4` package, dominated by a monolithic `dreamer4/dreamer4.py` (~3000 lines) that implements `VideoTokenizer`/`SpaceTimeTokenizer`, `DynamicsWorldModel`, `AudioTokenizer`, and `StateTokenizer`. These classes build causal space-time transformers with separate space/time attention blocks, latent flow-matching diffusion with shortcut consistency, BYOL-style tokenizer pre-training, PoPE attention, and a large zoo of experimental loss terms (latent autoregressive loss, decorrelation, orthogonality, SigReg, self-flow distillation, shortcut loss). `dreamer4/trainers.py` provides training loops (VideoTokenizerTrainer, BehaviorCloneTrainer, DreamTrainer, SimTrainer) and `dreamer4/cli.py` exposes CLI entry points for training and serving the world model. The repository is hybrid: primarily research code, but packaged and tested like a library. It ships a pyproject.toml with pinned dependencies, a parametrized test suite (tests/test_dreamer.py, test_e2e.py, test_env_wrapper.py), a browser-based Snake environment with a Flask-style server (`dreamer4/web_env/`) used for human demonstration collection and replay-buffer inspection, and environment wrappers in `dreamer4/env.py` that record agent experience to folders or memmap replay buffers. Root-level scripts are the actual experiment entry points, targeting standard benchmarks: HalfCheetah (imagination RL), CartPole (dynamics + RL), MovingMNIST (tokenizer + dynamics), and Snake (PPO with dynamics model). Craftsmanship is above typical research code: clean package layout, pytest coverage across many hyperparameter combinations, self-contained utilities (LossNormalizer, tensor pack/unpack helpers, with_seed decorator, memmap replay-buffer helpers), and HuggingFace Accelerate integration for distributed/mixed-precision training. However, the single-file core, 100+ hyperparameter flags on trainers, scattered hard-coded constants, sparse docstrings, and reliance on volatile niche packages (x-transformers, h-net-dynamic-chunking, PoPE_pytorch, adam-atan2-pytorch) make long-term stability and code reuse risky. The repository is recently committed and appears maintained, but it is explicitly beta-status experimental research code, not a production system.
FluxVLA is a comprehensive, config-driven framework for training, evaluating, and deploying Vision-Language-Action (VLA) models for robotic manipulation. It supports multiple VLA architectures (OpenVLA, Pi0/Pi0.5, GR00T, Cosmos3, DreamZero, etc.) and robot platforms (LIBERO simulation, RoboCasa, Franka, UR3, ALOHA, Oli humanoid, Tron2). The codebase provides a modular pipeline with registries and builders for datasets, transforms, models (backbones, heads, projectors), optimizers, collators, and inference runners. Production capabilities include distributed training (FSDP/DDP), real-time ROS inference, custom CUDA/Triton kernels for performance, and evaluation with metrics reporting to Feishu. While primarily an engineering framework for applied robot learning, it also contains research contributions such as flow-matching action heads, reward models (ARM, SARM), and real-time chunking (RTC) guidance. The architecture is clean and extensible, though config proliferation and vendored third-party code introduce maintenance overhead.
Evo-1 is a research codebase for a lightweight Vision-Language-Action (VLA) model accepted at CVPR 2026. It provides a complete pipeline for training (two-stage pretraining + finetuning using DeepSpeed and InternVL3-1B backbone) and evaluating robot manipulation policies on simulation benchmarks (Meta-World, LIBERO, LIBERO-Plus, RoboTwin) and real robots (xArm6, Aloha, SO100). The primary data flow involves converting raw robot datasets (LeRobot v2.1 format) into normalized state/action pairs with delta-timestep support, computing per-dataset norm statistics, and training a flow-matching action head with a frozen VLM backbone. The repo also includes a WebSocket-based inference server and clients for real-robot deployment. Although the core is research-oriented, it incorporates the full LeRobot framework for data collection and robot control, giving it a hybrid character. The codebase is well-structured with reusable components for dataset processing, normalization, and video handling, but the embedded LeRobot copy and multiple evaluation environments introduce maintenance overhead.
Musubi Tuner is a comprehensive LoRA training and inference toolkit for a wide range of diffusion model architectures, including HunyuanVideo, Wan, FramePack, FLUX variants, Qwen-Image, Z-Image, HiDream-O1, Ideogram4, Krea2, and Kandinsky5. It provides a unified pipeline of pre-caching (latents and text encoder outputs), training (LoRA, LoHa, LoKr), and generation, all accessible via CLI scripts and an optional Gradio GUI. The codebase is built on a shared `NetworkTrainer` base class with architecture-specific overrides, and includes advanced memory-saving techniques such as block-swap CPU offloading, FP8 quantization, and gradient checkpointing. While the repository is actively maintained and widely used (2035 stars), it exhibits medium technical drift due to vendored external model code, deprecated diffusers API usage, and a proliferation of near-duplicate architecture-specific scripts. The core reusable components—unified attention backend, offloading utilities, FP8 optimization, and LoRA network infrastructure—demonstrate solid engineering craftsmanship.
SimpleFold is a research repository focused on protein structure prediction using flow-matching generative models. While it includes application-like features such as a CLI and inference scripts, its core architecture is designed for ML research, specifically training and evaluating large-scale diffusion transformers (FoldingDiT) on structural biology data. The codebase is hybrid in nature, combining rigorous data engineering pipelines (parsing MMCIF, filtering, tokenization) with standard deep learning frameworks (PyTorch Lightning, Hydra). It supports dual backends (PyTorch and MLX) for inference, indicating a focus on accessibility across hardware platforms. The repository's primary value lies in its data processing pipeline, which is heavily adapted from the Boltz-1 project. It handles complex structural data (PDB, SwissProt, AFESM) through a modular pipeline including parsing, filtering, and tokenization. The model architecture implements flow-matching with ESM-2 embeddings for sequence context. Evaluation metrics include pLDDT confidence, TM-score, and RMSD, typical for structural biology research. However, the codebase exhibits signs of research-to-production transition challenges. Hardcoded paths (e.g., `/mnt/task_runtime/...`) and reliance on external binaries (TMscore, OpenStructure) reduce portability. The data pipeline is a fork of an external project (`boltz`), introducing maintenance overhead for divergence. Despite these issues, the use of Hydra for configuration management and PyTorch Lightning for training orchestration demonstrates solid engineering practices for an ML research project.
SkyReels-V3 is a multimodal video generation inference framework that supports three core tasks: reference-to-video (generating video from 1-4 reference images), video extension (single-shot and shot-switching), and talking avatar generation (from a portrait and audio). The codebase implements a diffusion transformer architecture (Wan-based) with custom components including a causal 3D VAE, T5 text encoder, CLIP image encoder, wav2vec2 audio encoder, and a flow-matching scheduler. It provides multi-GPU support via xDiT USP and low VRAM options with FP8 quantization and block offloading. The system is designed as a CLI tool with a single entry point (`generate_video.py`) that dispatches to task-specific pipelines. While primarily an application for running inference, it contains several reusable modules such as a custom scheduler, attention wrappers, and color transfer utilities.
Show-o and Show-o2 are research repositories for unified multimodal understanding and generation. They implement transformer-based models (1.3B, 1.5B, 7B parameters) that can perform text-to-image generation, image/video captioning, visual question answering, inpainting/extrapolation, and mixed-modality generation (interleaved text and images). The codebase is organized around two major model versions: Show-o uses discrete tokens with MAGVITv2 VAE and Phi-1.5 LLM, while Show-o2 moves to continuous latent space with WanVAE (3D causal VAE) and Qwen2.5 LLM. Training follows a multi-stage pipeline (pretraining, instruction tuning) with novel techniques like flow matching (via a modular `transport` package), dual-path vision encoders (SigLIP + patch embedding), and omni-attention masks that combine causal and bidirectional attention for multimodal sequences. The repository includes distributed training configs (DeepSpeed, multi-node), evaluation scripts for GenEval and DPG-Bench, and comprehensive YAML configuration files. This is primarily a data_research codebase: it produces research artifacts (models, papers at ICLR 2025 and NeurIPS 2025) rather than a production application. The code exhibits good modularity for a research project, with reusable components like the transport module, omni-attention masks, mixed dataloader, and attention masking utilities. However, it suffers from typical research-code issues: hardcoded internal infrastructure paths, reliance on non-public data pipelines (e.g., internal packages for RefinedWeb), and some code duplication between Show-o and Show-o2. Reproducibility is moderate—dependencies are pinned and configs are detailed, but many datasets are large-scale web sources that require manual setup, and the training scripts assume specific hardware (multi-node A100 clusters). The project is actively maintained (commits in early 2026) with a roadmap for future work, including mixed-modal generation and video support.