This repository implements generative recommendation models, primarily HSTU (Hierarchical Sequential Transduction Units) and DLRM-v3, for large-scale sequential recommendation. It is a hybrid codebase combining research experimentation (ICML'24 paper on generative recommenders) with production-grade inference benchmarking (MLPerf loadgen integration). The code provides custom high-performance CUDA and Triton kernels for flash attention, jagged tensor operations, and layer normalization, targeting modern NVIDIA GPUs (Hopper/Blackwell). It includes training pipelines with gin configuration for datasets like MovieLens, Amazon Books, KuaiRand, and synthetic streaming data up to 100B items. The repository also contains a full MLPerf inference benchmark suite with vendored loadgen library and end-to-end TorchScript tracing for deployment. While the core focus is research on scaling laws and architecture innovations for sequential recommenders, the ops and modules are engineered to production standards with multiple kernel backends and rigorous tests.
FlagAttention is a library implementing memory-efficient attention operators in the Triton language. It provides optimized implementations of flash attention, piecewise attention, paged attention, and gated delta rule operations, designed for use in large language model training and inference. The library features modular architecture with separate implementations for each attention variant, comprehensive testing, and example usage patterns. It leverages Triton for GPU-accelerated computations while maintaining compatibility with PyTorch's autograd system.
LongCat-Video is a production-grade video generation foundation model (13.6B parameters) that supports text-to-video, image-to-video, video continuation, and audio-driven avatar animation. The codebase implements a dense DiT-style transformer with 3D patch embedding, block-sparse attention (BSA) via custom Triton kernels, 3D rotary position embeddings (RoPE), and context parallelism for multi-GPU inference. It includes a coarse-to-fine generation pipeline (480p → 720p) using LoRA adapters for distillation and refinement, and an avatar variant that conditions on audio embeddings from Wav2Vec2 or Whisper. The architecture is monolithic, with entry points being demo scripts and a Streamlit UI. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, Triton, and optional OpenAI API for prompt enhancement. The code demonstrates advanced engineering practices such as INT8 quantization, KV-caching, and tiled VAE decoding, but suffers from some code duplication and tight coupling to specific model dimensions.
This repository is the official implementation of the SANA family of diffusion models for high-resolution image and video generation, developed by NVIDIA Research. It is primarily a research codebase, containing training and inference pipelines for multiple model variants: SANA (text-to-image), SANA-1.5, SANA-Sprint (few-step distillation), SANA-Video, SANA-Video 2.0, SANA-Streaming (long video), LongSANA, and SANA-WM (world model for camera-controlled video). The codebase also includes post-training with reinforcement learning (Sol-RL), ControlNet support, and various schedulers (flow matching, DPM-Solver, LCM, self-forcing). While the core purpose is research, the repository also provides deployable Gradio applications, conversion scripts to Hugging Face diffusers format, and integration with ComfyUI and SGLang, giving it a hybrid application character. The code is heavily optimized with custom Triton kernels for efficient attention (LiteMLA, Gated Delta Net) and includes distributed training utilities (FSDP, context parallelism). The repository is actively maintained with frequent commits and supports a wide range of model sizes (600M to 14B parameters).