$ show meituan-longcat/LongCat-Video
LongCat-Video is a production-grade video generation foundation model (13.6B parameters) that supports text-to-video, image-to-video, video continuation, and audio-driven avatar animation. The codebase implements a dense DiT-style transformer with 3D patch embedding, block-sparse attention (BSA) via custom Triton kernels, 3D rotary position embeddings (RoPE), and context parallelism for multi-GPU inference. It includes a coarse-to-fine generation pipeline (480p → 720p) using LoRA adapters for distillation and refinement, and an avatar variant that conditions on audio embeddings from Wav2Vec2 or Whisper. The architecture is monolithic, with entry points being demo scripts and a Streamlit UI. External dependencies include PyTorch, HuggingFace Transformers, FlashAttention, Triton, and optional OpenAI API for prompt enhancement. The code demonstrates advanced engineering practices such as INT8 quantization, KV-caching, and tiled VAE decoding, but suffers from some code duplication and tight coupling to specific model dimensions.
The LLM identified 4 files worth highlighting for reuse:
Provides a generic LoRA injection framework that can wrap any Linear layer with low-rank adapters. Supports multiple LoRA adapters simultaneously (e.g., for distillation and refinement) and handles forward passes with adapter selection. The implementation is self-contained and could be extracted for use in other transformer-based models.
Implements per-channel symmetric weight-only INT8 quantization for Linear layers, with support for loading sharded safetensors. The quantization is applied at inference time and includes dequantization during forward pass. This module is reusable for any PyTorch model requiring memory-efficient inference.
Defines the interface for block sparse attention (BSA) using custom Triton kernels. It includes mean-pooling gating, top-k or CDF-based sparse mask selection, and integration with FlashAttention. The implementation is modular and could be adapted for other transformer architectures needing efficient sparse attention for long sequences.
Contains the VAE decoder with 3D causal convolutions (WanCausalConv3d) that support feature caching for efficient frame-by-frame inference. The modular design (residual blocks, attention blocks, resampling) allows reuse in other video generation or compression models. The caching mechanism is a reusable pattern for sliding-window temporal processing.