AlphaFold3 is Google DeepMind's official command-line application for running AlphaFold 3 biomolecular structure prediction inference. The dominant character is a production-grade scientific application, though it is heavily intertwined with research data pipelines: it ingests a JSON job specification describing protein, RNA, DNA, and ligand chains, runs Jackhmmer/Nhmmer searches over large genetic databases, builds multiple sequence alignments, searches structural templates with HMMER, featurizes the input into JAX/Haiku tensors, samples structures with a diffusion model, and writes predicted structures as mmCIF files plus confidence JSON (pLDDT, PAE, pTM, ipTM). The codebase is not a training repository; it provides inference only and requires separately downloaded model parameters under restrictive legal terms. Architecturally, the system is a multi-stage pipeline driven by the `run_alphafold.py` entry point. Core modules include the data pipeline (`src/alphafold3/data/pipeline.py`), MSA/template tools (`src/alphafold3/data/tools/`), a structure representation layer (`src/alphafold3/structure/`), JAX geometric primitives (`src/alphafold3/jax/geometry/`), the neural network model (`src/alphafold3/model/`), and performance-critical C++ pybind11 extensions for mmCIF parsing, MSA feature extraction, and string/array operations. It depends on external binaries (Jackhmmer, Nhmmer, HMMER suite), RDKit for ligand conformer generation, JAX/Haiku for model execution, and ~630 GB of downloadable sequence and structure databases. The repository is actively maintained, with recent commits and CI running CPU-only data tests. Technical drift is low for a research release: dependencies are pinned via `pyproject.toml` and `uv.lock`, and active maintenance keeps it aligned with its ecosystem, though the C++ build, GPU-specific requirements, and external database URLs remain fragility points.
Protenix is a production-oriented PyTorch implementation of AlphaFold 3 for biomolecular structure prediction, maintained by ByteDance. It is best classified as an application rather than a pure research codebase: it ships a `protenix` CLI with commands for prediction, input conversion, MSA/template/RNA-MSA search, and data preparation, plus training/fine-tuning entry points, batch inference runners, Docker deployment, web-service helpers, and PyPI packaging. The code ingests PDB/CIF, FASTA, SMILES/SDF, and JSON descriptors covering proteins, DNA, RNA, ligands, ions, modified residues, covalent bonds, and contact/pocket constraints, then outputs predicted 3D CIF structures with confidence metrics such as pLDDT, PAE, pTM, and ipTM.
HY-WU is a machine learning application for text-guided image editing that implements a Functional Neural Memory framework. The system generates instance-conditioned LoRA (Low-Rank Adaptation) weights on-the-fly from hybrid image-instruction representations and injects them into a frozen backbone model (HunyuanImage-3.0-Instruct) during forward pass, enabling test-time adaptation without optimization. The architecture centers around a WUPipeline orchestrator that coordinates a parameter generator, the base model with PG mixin integration, and image processing components. Key entry points include the infer.py CLI script and gradio/app.py web interface. The system processes user prompts and input images through VAE encoding, condition extraction, parameter generation, LoRA injection, and image decoding to produce edited outputs.
AudioX-Turbo is a unified framework for efficient anything-to-audio generation (text, video, audio conditions → audio) using Distribution Matching Distillation (DMD) adapted to flow matching and adversarial training. It supports six core tasks: Text-to-Audio, Text-to-Music, Video-to-Audio, Video-to-Music, Text+Video-to-Audio, and Text+Video-to-Music. The system includes training scripts, an interactive Gradio demo, and a modular architecture with interchangeable components (attention, bottlenecks, etc.) via a factory pattern. Key innovations enable high-quality 4-step audio generation, reducing computational cost by up to 25× compared to multi-step diffusion models.
Iris is a high-performance C inference engine for diffusion image generation models, specifically designed for FLUX.2 Klein (4B/9B) and Z-Image-Turbo (6B) architectures. It provides a complete pipeline from text encoding (Qwen3-4B/8B) through transformer denoising to VAE decoding, outputting PNG or PPM images. The system operates as both a standalone CLI tool with an interactive REPL and a C library (libiris.a), featuring memory-mapped weight loading for efficiency (~4-5GB peak for 4B model) and optional Metal GPU acceleration on Apple Silicon for significant speedups. The architecture is modular: separate components handle safetensors weight loading, BPE tokenization, transformer blocks (double/single stream for Flux, main/refiner for Z-Image), VAE decoding, and image I/O (PNG/JPEG/PPM). The CLI supports text-to-image, image-to-image, multi-reference generation, and various scheduling strategies (shifted sigmoid, linear, power). Memory optimization techniques include releasing the text encoder after embedding extraction, embedding caching with 4-bit quantization, and dynamic attention memory budgeting for reference images. The codebase demonstrates deep systems engineering and ML inference expertise, with hand-optimized kernels, custom Metal shaders, and detailed performance instrumentation. It is self-contained with no external dependencies beyond standard C libraries and optional BLAS/Metal, making it portable across Linux and macOS. The project includes regression test vectors and a JPEG decoder fuzzer, indicating a focus on correctness and reliability.
LongCat-Video is a production-grade video generation foundation model (13.6B parameters) that supports text-to-video, image-to-video, video continuation, and audio-driven avatar animation. The codebase implements a dense DiT-style transformer with 3D patch embedding, block-sparse attention (BSA) via custom Triton kernels, 3D rotary position embeddings (RoPE), and context parallelism for multi-GPU inference. It includes a coarse-to-fine generation pipeline (480p → 720p) using LoRA adapters for distillation and refinement, and an avatar variant that conditions on audio embeddings from Wav2Vec2 or Whisper. The architecture is monolithic, with entry points being demo scripts and a Streamlit UI. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, Triton, and optional OpenAI API for prompt enhancement. The code demonstrates advanced engineering practices such as INT8 quantization, KV-caching, and tiled VAE decoding, but suffers from some code duplication and tight coupling to specific model dimensions.
Irodori-TTS is a state-of-the-art Japanese text-to-speech system built on rectified flow diffusion over DACVAE audio latents. It provides a complete training and inference stack: `train.py` supports full fine-tuning, LoRA, speaker inversion, and MeanFlow distillation with distributed training and WSD scheduling; `infer.py` offers a rich CLI with CFG, Sway Sampling, duration scaling, and watermarking; `gradio_app.py` and `gradio_app_voicedesign.py` expose interactive UIs for reference-based cloning and caption-conditioned voice design. The architecture is modular (`irodori_tts` package) with attention dispatch (FA3/SDPA), a hybrid Muon+AdamW optimizer, codec abstraction, and caching-friendly `InferenceRuntime`. While highly engineered for production use (INT4/INT8 quantization, safetensors, HuggingFace Hub integration), it remains a research-oriented codebase with experimental sampling algorithms and many YAML configs for different model sizes/versions. The code is clean and well-factored, though `train.py` is a large monolith and external dependencies (DACVAE, silentcipher) are pinned to specific commits, creating ecosystem fragility.
Alpamayo 1 is a Vision-Language-Action (VLA) model designed for autonomous driving applications. It implements the Alpamayo-R1 architecture that combines a Qwen3-VL backbone for vision-language reasoning with a diffusion-based action expert for trajectory prediction. The system takes multi-camera video and egomotion history as inputs, generates Chain-of-Causation (CoC) reasoning traces, and outputs future trajectories (6.4s horizon at 10Hz). Key components include a unicycle kinematic model for action space conversion, CUDA graph acceleration for efficient repeated inference, and robust geometric transformations. While the repository is noted to receive only limited maintenance as newer versions exist, the current implementation demonstrates strong engineering practices with several reusable components.
MAGREF is a video generation application that creates videos from text prompts and reference images using a diffusion model with subject disentanglement. The system implements a modular architecture separating model logic, configurations, distributed computing support, and utility functions. Key entry points include generate.py (main script) and shell scripts for single/multi-GPU inference. The primary data flow involves encoding text prompts with T5, processing reference images through CLIP and custom resizing/padding, running diffusion steps with a WanModel transformer, and decoding latents to video via VAE. External dependencies include PyTorch, Hugging Face Transformers and Diffusers, xfuser for context parallelism, and flash-attention for efficient computation.
Chai-1 is a molecular structure prediction tool that takes FASTA sequences (proteins, DNA, RNA, ligands, glycans) and predicts 3D structures using a diffusion model. It is packaged as a Python library with a CLI (`chai-lab fold`) and a Python API (`chai_lab.chai1.run_inference`). The pipeline ingests sequences, optionally generates MSAs via ColabFold server, loads templates from RCSB, computes ESM embeddings, applies user-defined restraints (contacts, pockets, covalent bonds), tokenizes all atoms, generates a rich set of features, collates them into batches, and runs a TorchScript diffusion model to produce multiple structure samples. Outputs are ranked by confidence metrics (pTM, ipTM, pLDDT, clash score) and saved as CIF files. The codebase is well-structured with clean abstractions (feature generators, data contexts, collation), strong typing (jaxtyping, beartype), and CI quality gates. It is a hybrid of a production-grade inference tool and research code, with some copied components (rigid transformations from AlphaFold, ColabFold MSA server) and a monolithic inference entry point. The repository is actively maintained with recent commits, PyPI releases, and comprehensive examples.
BoltzGen is a deep learning pipeline for computational protein and peptide binder design. It provides a command-line interface (boltzgen) that orchestrates a multi-step workflow: diffusion-based backbone generation, inverse folding (sequence design given structure), folding with Boltz-2, affinity prediction, analysis, and filtering/ranking of designs. The system supports diverse protocols including protein, peptide, nanobody, antibody, and small-molecule targeting, with extensive conditioning options like binding sites, disulfide bonds, cyclization, and per-residue constraints. The codebase is structured as a Python package with modular components for data parsing (YAML design specs, mmCIF/PDB), featurization, tokenization, cropping, and model layers (pairformer, attention, diffusion, inverse folding). It includes training pipelines with Hydra configuration, PyTorch Lightning, and validation splits. While primarily an application, it also contains research-grade example datasets and a Jupyter notebook for filtering, reflecting its origin in a bioRxiv publication. The architecture is well-organized but shows some technical drift (hardcoded paths, print statements, broad exception handling) and dependency on specific CUDA versions.
Helios is a real-time long video generation system that implements text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation using a 3D transformer architecture. The system features a modular design with specialized optimizations for performance including Ahead-of-Time Compilation (AOTI), group offloading for VRAM reduction, context parallelism (Ulysses/Ring attention), and Flash Attention kernel optimizations. It employs a multi-stage training pipeline (Base → Mid → Distilled) to balance quality and sampling speed. The repository includes a Gradio web interface (app.py) for interactive generation, command-line inference scripts, training utilities, and a comprehensive evaluation suite measuring video quality across aesthetic, motion, semantic, and naturalness dimensions. While containing research elements for ablation studies and benchmarking, its primary function is as a deployable video generation application.
The repository implements LLaVA-VLA, a Vision-Language-Action model for robotic manipulation tasks. It extends the LLaVA framework to predict action sequences from visual inputs, proprioceptive state, and language instructions using a diffusion-based action head. The system includes training pipelines for fine-tuning on datasets like CALVIN, inference servers (Flask and Gradio) for real-time robot control, and evaluation tools for benchmarks. The architecture features modular vision encoders, projectors, and resamplers with factory patterns for flexibility, action tokenization for discrete action prediction, and integration with PEFT/DeepSpeed for efficient training.
VoxCPM is a tokenizer-free, multilingual text-to-speech (TTS) system developed by OpenBMB. It supports voice cloning, voice design, and controllable speech generation across 30 languages and 9 Chinese dialects, outputting 48kHz audio. The architecture is a diffusion-autoregressive model built on MiniCPM-4, comprising AudioVAE, Local Encoder, Text-Semantic LM, Residual Acoustic LM, and Local DiT. The repository provides multiple interfaces: a Gradio web application (app.py) for interactive use, a command-line interface (voxcpm.cli) for batch processing, a fine-tuning web UI (lora_ft_webui.py) for LoRA and full-parameter training, and Python APIs for integration. It also includes utilities for audio enhancement (ZipEnhancer), timestamp alignment, and text normalization. While primarily an application, it contains research-grade model definitions and training pipelines, making it a hybrid project with a strong application focus.
EchoMimicV3 is a unified multi-modal and multi-task human animation model that generates realistic human videos from audio and/or text prompts. Built upon the Wan2.1-Fun-V1.1-1.3B-InP base model, it supports both talking-head and full-body animation with two versions: preview (standard quality) and flash (optimized for speed and lower VRAM). The system provides Gradio-based interactive demos (app.py, app_mm.py) for real-time video generation from user-uploaded media, command-line inference scripts (infer_flash.py, infer_preview.py, run_flash.sh) for batch processing, and deployment options for ComfyUI, ModelScope, and HuggingFace. Core functionality centers on audio-driven lip-sync and facial animation, with optional text guidance for pose/scene control, leveraging diffusion models, transformer architectures, and multimodal encoders (CLIP for images, T5 for text, Wav2Vec2 for audio).
IgGM is a generative foundation model for antibody and nanobody design, implemented as a deep learning library with a command-line interface. The system performs tasks such as CDR loop design, framework region redesign (humanization), inverse design from complex structures, affinity maturation, and structure prediction. It uses a diffusion-based approach to co-design amino acid sequences and 3D backbone structures, leveraging an ESM2-based protein language model, Evoformer blocks with triangular attention, invariant point attention, and a structure module similar to AlphaFold2. The pipeline is orchestrated by the `AbDesigner` class and invoked via the `design.py` script, accepting PDB and FASTA inputs and generating output sequences and structures. The repository includes well-structured reusable components for rigid-body transformations (Rotation, Rigid), PDB parsing, coordinate conversion, and diffusion schedules, though some research artifacts (misspellings, commented code) and dependency pinning to older PyTorch/CUDA versions indicate moderate technical drift.
This repository implements VTP (Visual Tokenizer Pre-training), a framework for pre-training visual tokenizers using a combination of contrastive, self-supervised, and reconstruction learning objectives. The system provides a unified model architecture that supports CLIP-style image-text contrastive learning, DINOv2-style self-supervised learning, and image reconstruction. It includes training scripts, evaluation tools for zero-shot classification, linear probing, and reconstruction metrics, as well as integration with LightningDiT for generation evaluation. The codebase features a HuggingFace-compatible model implementation, modular transformer components, and various utility functions for image processing and metric calculation.
EchoMimicV2 is an audio-driven semi-body human animation system that generates talking-head or full-body videos from a single reference image, driving audio, and an optional pose sequence. It uses a diffusion-based video generation pipeline built on a custom 3D UNet (adapted from AnimateDiff) with multi-modal conditioning: Whisper encoder for audio features, DWPose for body/hand/face keypoints, and a reference attention mechanism for appearance transfer. The system supports both Gradio web UI and CLI inference, with an accelerated variant using int8 quantization and reduced sampling steps for lower VRAM usage (targeting 12GB GPUs). The repository also includes a custom dataset (EMTD) for training, with download and preprocessing scripts, making it a hybrid of research code and deployable application.