Open-OmniVCus is a research monorepo reproducing the NeurIPS 2025 OmniVCus method for subject-driven video customization under multimodal control conditions (reference image, depth, mask, motion/camera). It is hybrid by nature: the dominant character is an application/framework layer provided by a large vendored/forked copy of DiffSynth-Studio, wrapped around research-oriented data-construction and training code (VideoCus-Factory, Video-Depth-Anything). The repo does not expose a hosted service or API; it is a Python library plus executable training/inference scripts for diffusion-based image and video generation.
This repository is the official implementation of the SANA family of diffusion models for high-resolution image and video generation, developed by NVIDIA Research. It is primarily a research codebase, containing training and inference pipelines for multiple model variants: SANA (text-to-image), SANA-1.5, SANA-Sprint (few-step distillation), SANA-Video, SANA-Video 2.0, SANA-Streaming (long video), LongSANA, and SANA-WM (world model for camera-controlled video). The codebase also includes post-training with reinforcement learning (Sol-RL), ControlNet support, and various schedulers (flow matching, DPM-Solver, LCM, self-forcing). While the core purpose is research, the repository also provides deployable Gradio applications, conversion scripts to Hugging Face diffusers format, and integration with ComfyUI and SGLang, giving it a hybrid application character. The code is heavily optimized with custom Triton kernels for efficient attention (LiteMLA, Gated Delta Net) and includes distributed training utilities (FSDP, context parallelism). The repository is actively maintained with frequent commits and supports a wide range of model sizes (600M to 14B parameters).
Helios is a real-time long video generation system that implements text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation using a 3D transformer architecture. The system features a modular design with specialized optimizations for performance including Ahead-of-Time Compilation (AOTI), group offloading for VRAM reduction, context parallelism (Ulysses/Ring attention), and Flash Attention kernel optimizations. It employs a multi-stage training pipeline (Base → Mid → Distilled) to balance quality and sampling speed. The repository includes a Gradio web interface (app.py) for interactive generation, command-line inference scripts, training utilities, and a comprehensive evaluation suite measuring video quality across aesthetic, motion, semantic, and naturalness dimensions. While containing research elements for ablation studies and benchmarking, its primary function is as a deployable video generation application.
GeekAI is a full-stack, production-grade AI creation platform providing chat, image generation (MidJourney, DALL-E, Jimeng, Stable Diffusion), video generation (Sora, Veo, Keling, MiniMax, Wan, Doubao), music generation (Suno), and PPT creation. The system features a Go + Gin backend with MySQL/Redis persistence and a Vue 3 (Element Plus/Vant) frontend supporting both desktop and mobile interfaces. It includes a complete monetization framework with power credits, multiple payment gateways (Alipay, WeChat Pay, EPay), user invite/reward system, and an administrative dashboard for managing users, models, API keys, and system configurations. Architecturally, the backend follows a layered pattern with handlers, services, and stores. Key design patterns include an adapter pattern for video generation providers (allowing easy addition of new AI services), a factory pattern for OSS uploaders (local, MinIO, Qiniu, Aliyun, Tencent), and an abstraction for payment integrations. Async tasks (e.g., image generation, video creation) are handled via Redis-based queues with polling for status updates. The frontend uses Pinia stores for state management and includes reusable UI components like custom tabs, file uploaders, and parameter builders. The codebase demonstrates solid engineering with moderate code quality. It has several genuinely reusable patterns (video adapter, upload manager, payment service interface) but also exhibits duplication between desktop and mobile views, some hardcoded configuration, and reliance on many third-party SDKs that may drift over time. While not exceptionally polished, it is a fully functional commercial-grade platform suitable for deployment and customization.
This repository is a ComfyUI custom node pack (ComfyUI-dapaoAPI) that provides a comprehensive suite of nodes for AI-powered image, video, audio, and text generation. It acts as a unified relay client to multiple commercial AI services (dapaoAI, RunningHub, Gemini, Seedance, Wan, etc.), abstracting API details, handling media preprocessing, error diagnostics, task queue persistence, and concurrent batch execution. The architecture is a classic ComfyUI extension: each node type is defined as a Python class with input/output specifications, and shared utility modules (e.g., `dreamshark_runtime` for job submission, `image_input_utils` for tensor-to-PNG conversion, `node_error_utils` for formatted error handling) are used across all generation nodes. A front-end JavaScript layer adds dynamic widget behavior, price badges, and rich prompt editors within ComfyUI's existing UI framework. The pack notably includes a hybrid nature: it bundles a `skills/` directory containing detailed markdown-based AI agent instructions (for short drama writing, music caption rewriter, article formatting, GPT image templating) that are injected into LLM prompts during node execution or used as standalone agent guides. These skills are not data pipelines but rather structured knowledge assets for prompt engineering, giving the repository a dual character as both a custom node collection and an LLM orchestration framework.
This repository is a collection of standalone examples demonstrating the MLX framework (Apple's machine learning framework for Apple Silicon). It covers a wide range of deep learning models including LLMs (Mistral, Mixtral, Llama, T5), image generation (Stable Diffusion, FLUX, CLIP), video generation (Wan2.1), audio (Whisper, MusicGen, EnCodec), segmentation (SAM), and more. Each example is self-contained with its own entry point (CLI or Python script), model conversion utilities, and often a training/fine-tuning script. The repository is primarily an application-focused showcase with some research elements (e.g., normalizing flows, GCN). Production entry points include `whisper/mlx_whisper/cli.py` for transcription, `flux/txt2image.py` for image generation, `video/wan2.1/txt2video.py` for video generation, and `lora/lora.py` for fine-tuning. All examples depend on the MLX library and often bundle conversion from PyTorch weights. The architecture is a flat collection of independent projects, not a unified framework, leading to some code duplication (e.g., T5 and EnCodec are copied into multiple directories). However, several components are well-crafted and reusable, notably the Whisper decoding pipeline, the LoRA adapter module, and the flow-matching samplers for diffusion models.
HunyuanVideo-1.5 is an open-source video generation model by Tencent, implemented as a Python library with CLI scripts for inference (`generate.py`) and training (`train.py`). It generates high-quality videos from text or image prompts using a diffusion transformer (DiT) architecture with 8.3B parameters, supporting resolutions up to 1080p via super-resolution. The codebase provides a modular pipeline (`HunyuanVideo_1_5_Pipeline`) that orchestrates prompt encoding (byT5 text encoder, SigLIP vision encoder), denoising with flow matching, and VAE decoding. It includes advanced features like SSTA (Selective and Sliding Tile Attention) for efficient long-video processing, step distillation, CFG distillation, and various caching mechanisms (DeepCache, TeaCache, TaylorCache). The repository also offers a training script with distributed support (FSDP, context parallelism), LoRA fine-tuning, and the Muon optimizer. While primarily an application for video generation, it contains research-grade model implementations and evaluation benchmarks. The code is well-structured with reusable neural network modules, but relies on many optional hardware-specific dependencies (flash-attn, flex-block-attn, SageAttention) and external services for prompt rewriting, which may introduce operational complexity. The license is a custom Tencent Hunyuan Community License with restrictions.
Musubi Tuner is a comprehensive LoRA training and inference toolkit for a wide range of diffusion model architectures, including HunyuanVideo, Wan, FramePack, FLUX variants, Qwen-Image, Z-Image, HiDream-O1, Ideogram4, Krea2, and Kandinsky5. It provides a unified pipeline of pre-caching (latents and text encoder outputs), training (LoRA, LoHa, LoKr), and generation, all accessible via CLI scripts and an optional Gradio GUI. The codebase is built on a shared `NetworkTrainer` base class with architecture-specific overrides, and includes advanced memory-saving techniques such as block-swap CPU offloading, FP8 quantization, and gradient checkpointing. While the repository is actively maintained and widely used (2035 stars), it exhibits medium technical drift due to vendored external model code, deprecated diffusers API usage, and a proliferation of near-duplicate architecture-specific scripts. The core reusable components—unified attention backend, offloading utilities, FP8 optimization, and LoRA network infrastructure—demonstrate solid engineering craftsmanship.
EchoMimicV3 is a unified multi-modal and multi-task human animation model that generates realistic human videos from audio and/or text prompts. Built upon the Wan2.1-Fun-V1.1-1.3B-InP base model, it supports both talking-head and full-body animation with two versions: preview (standard quality) and flash (optimized for speed and lower VRAM). The system provides Gradio-based interactive demos (app.py, app_mm.py) for real-time video generation from user-uploaded media, command-line inference scripts (infer_flash.py, infer_preview.py, run_flash.sh) for batch processing, and deployment options for ComfyUI, ModelScope, and HuggingFace. Core functionality centers on audio-driven lip-sync and facial animation, with optional text guidance for pose/scene control, leveraging diffusion models, transformer architectures, and multimodal encoders (CLIP for images, T5 for text, Wav2Vec2 for audio).
VideoCoF is a unified video editing framework based on diffusion transformers (Wan2.1). It implements a 'See → Reason → Edit' pipeline using chain-of-frames (CoT) reasoning, enabling object removal, addition, swap, and local style transfer. The architecture includes a modular pipeline (`WanPipeline`) with flow matching solvers, VAE, text encoder, and transformer backbone. Inference can be done via a fast inference script (`fast_infer.py`) using 4-step DMD, batch inference via JSON, or through a Gradio UI. Training supports LoRA fine-tuning with DeepSpeed ZeRO-2/3 for 1.3B and 14B models, with custom dataset classes (including CoT and joint image-video). External dependencies include Diffusers, Transformers, Gradio, OpenCV, DeepSpeed, and optional paifuser for optimizations. While primarily an application for video editing, the repo has strong research character with extensive evaluation metrics (CLIP, GPT), configuration files, and experiment scripts. The code is modular in parts but contains experimental features, hardcoded paths, and reliance on an internal optimization package (paifuser) that may limit external reuse. The core utility modules (LoRA, caching, attention, schedulers) are well-crafted and potentially reusable across other diffusion model projects.
SkyReels-V3 is a multimodal video generation inference framework that supports three core tasks: reference-to-video (generating video from 1-4 reference images), video extension (single-shot and shot-switching), and talking avatar generation (from a portrait and audio). The codebase implements a diffusion transformer architecture (Wan-based) with custom components including a causal 3D VAE, T5 text encoder, CLIP image encoder, wav2vec2 audio encoder, and a flow-matching scheduler. It provides multi-GPU support via xDiT USP and low VRAM options with FP8 quantization and block offloading. The system is designed as a CLI tool with a single entry point (`generate_video.py`) that dispatches to task-specific pipelines. While primarily an application for running inference, it contains several reusable modules such as a custom scheduler, attention wrappers, and color transfer utilities.
This repository is a curated literature review (awesome list) focused on the progression from video generation to world models in AI research. It systematically organizes and categorizes recent academic papers (primarily from 2022-2026) into a hierarchical structure covering technical foundations, application domains, evaluation methodologies, and theoretical perspectives. The main value lies in its comprehensive paper collection with structured metadata including publication dates, venues, paper links, project pages, and official code repositories. The repository follows a standard awesome-list format using markdown tables with shields.io badges for easy navigation to paper resources. Content is organized into eight main sections: Generation 1-4 (covering faithfulness, interactiveness, planning, and randomness aspects), Evaluation and Datasets, Study and Rethinking, Downstream Tasks, and World Models for Other Applications. Each section contains subsections detailing specific technical approaches (e.g., GAN-based, U-Net-based, DiT-based video generation) and application areas (robotics, autonomous driving, gaming). The list is actively maintained with frequent updates incorporating very recent pre-print papers. As a static reference resource, the repository contains no executable code, data processing pipelines, or computational components. Its reproducibility is high because the curated content is directly available in the README upon cloning, requiring no setup or dependencies to access the paper list. The organizational pattern demonstrates good practices for academic literature curation, though it does not contain novel reusable data engineering components beyond the standard markdown documentation format.
The repository provides a comprehensive framework for generative AI models, particularly focused on image and video generation. It integrates various diffusion-based architectures (Stable Diffusion, Stable Diffusion XL, Hunyuan DiT, Wan, etc.) and offers utilities for model loading, prompt engineering, and post-processing. The system supports text-to-image, text-to-video, image-to-video, and video-to-video generation, along with editing capabilities via ControlNet and LoRA. The architecture is modular, with core components for model management, prompting, and VRAM optimization, though it exhibits some duplication across model-specific pipelines.
This repository is a collection of AI agent skill definitions for the Higgsfield platform, enabling automated content creation across multiple domains including brand identity, product photography, website generation, video explainers, YouTube thumbnails, and 3D game assets. Each skill is self-contained in a folder with a SKILL.md entry point and references directory, designed for consumption by AI agents (Claude, Cursor, Codex). The repo also includes deterministic scripts for GLB processing, texture pipeline, procedural rigging, and brandbook generation, as well as CI validation workflows. While primarily a skill library, it contains reusable engineering patterns such as a brandkit state machine, stdlib-only GLB patching, and a texture seam-fix pipeline. The codebase is well-structured with version sync enforcement and anti-slop design rules, but relies on external tools (Blender, ImageMagick, LibreOffice) and live Higgsfield API endpoints.
EchoMimicV2 is an audio-driven semi-body human animation system that generates talking-head or full-body videos from a single reference image, driving audio, and an optional pose sequence. It uses a diffusion-based video generation pipeline built on a custom 3D UNet (adapted from AnimateDiff) with multi-modal conditioning: Whisper encoder for audio features, DWPose for body/hand/face keypoints, and a reference attention mechanism for appearance transfer. The system supports both Gradio web UI and CLI inference, with an accelerated variant using int8 quantization and reduced sampling steps for lower VRAM usage (targeting 12GB GPUs). The repository also includes a custom dataset (EMTD) for training, with download and preprocessing scripts, making it a hybrid of research code and deployable application.
This repository provides custom nodes for ComfyUI to enable Deforum-based animation generation. It implements frame-by-frame processing of stable diffusion models with parameter scheduling, allowing users to create animated sequences by evolving prompts, seeds, strength, and other parameters over time. The architecture centers around ComfyUI's node system, with specialized nodes for parameter scheduling, prompt engineering, sampling, latent caching, ControlNet integration, image processing, logic control, video I/O, audio synchronization, and advanced noise generation. A critical execution hijack mechanism modifies ComfyUI's internal processing to handle list operations for frame sequences. The system relies on external libraries like Deforum Studio, ControlNet, RAFT, depth models, and FILM for computer vision techniques.
This repository is a curated awesome-list and documentation hub for the LTX-2/2.3/2.5 video generation suite, not a runnable software application. It aggregates downloadable checkpoints, GGUF quantizations, LoRAs, text encoders, ComfyUI node packs, training tools, and workflows from Hugging Face, GitHub, and community sources. The repository contains no application code, tests, CLI, or data pipeline; its primary deliverable is structured, link-heavy markdown content that serves as an index and practical reference for people working with LTX models in ComfyUI and local inference environments. Because it is not a data/research pipeline, it is classified as 'application' only in the sense that it is a content/knowledge artifact rather than a data project. The structure is modular: the README is effectively assembled from blocks/ files (00_header through 99_footer) that map to sections such as apps, models, text encoders, components, LoRAs, nodes, training, and workflows. A guides/ directory contains prompt-engineering and settings guides, including a detailed LTX-2 mastering guide and a basics guide covering FPS, resolutions, VRAM flags, and the two-stage generation pipeline using latent upscalers and distilled LoRAs. The content conventions—standardized model/precision/size tables, badge link definitions, drill-down details blocks, and explicit compatibility caveats—make the maintenance model clear and partially reusable. The main risks are external-link fragility, fast model-version churn across LTX-2, 2.3, and 2.5, and the need for custom forks for some quantization formats. There are also significant NSFW/uncensored content sections, gated model links, and several suspicious future-dated arXiv references that may be unverifiable. As a result, downstream users should not blindly trust every listing without checking the original sources, but the overall curation quality is solid for a community awesome-list.
This repository implements Geometry Forcing, a research framework for video generation with geometric consistency. The core contribution marries video diffusion models (DFoT - Diffusion Forcing Transformer) with 3D representation learning (VGGT) to produce temporally consistent videos conditioned on camera poses. It includes full training pipelines (Hydra + PyTorch Lightning), a suite of video metrics (FVD, FID, LPIPS, VBench), and evaluation scripts for reprojection/revisit error using DROID-SLAM. The codebase is structured as a research template with reusable base classes for algorithms, data modules, experiments, and metric registries. Raw datasets (RealEstate10K, Kinetics-600, Minecraft) are downloaded and preprocessed into video clips with associated camera parameters. Transformations include VAE encoding for latent diffusion, augmentation (flip, reverse), and sliding-window clip extraction. The primary research question is whether video diffusion can be constrained by explicit 3D geometry to reduce inconsistencies in generated videos, measured by geometry-aware metrics. The repo also demonstrates composable sampling via History Guidance and an alignment loss that projects diffusion latents into VGGT feature space. While the code is well-engineered for research (modular design, config-driven, metric registry), it carries typical research code baggage: hardcoded paths, vendored third-party code (DROID-SLAM, VGGT), potential dependency pinning conflicts, and some dead/commented code. Reproducibility is moderate: dependencies are pinned but no environment definitions beyond requirements.txt, and dataset downloads rely on fragile YouTube scraping.