$ show meituan-longcat/LongCat-Flash-Omni
LongCat-Flash-Omni is an application providing inference capabilities for a 560B parameter omni-modal model that processes text, audio, image, and video inputs to generate text and/or audio outputs. The system features a modular architecture with separate modality encoders (audio, vision), a unified data processor for multimodal inputs, and a post-processor for converting model outputs to usable formats. The main entry point is longcat_omni_demo.py which orchestrates the inference pipeline, leveraging SGLang for efficient model serving. The repository demonstrates strong engineering practices in its modular design, particularly in the audio feature processing and image transformation components.
The LLM identified 3 files worth highlighting for reuse:
Implements a GPU-accelerated audio feature processing pipeline including FBANK, CMVN, splicing, and delta computation. The FeatureProcessor class combines these components into a reusable pipeline that could be extracted for use in any audio processing project. Demonstrates non-obvious skill in optimizing these operations for batch processing on GPU.
Contains the LongCatImageTransform class and smart_resize function that implement aspect-ratio preserving image resizing with pixel constraints. This component is self-contained, demonstrates sophisticated computer vision techniques, and could be easily reused in other image or video processing projects.
Provides a clean separation of concerns for converting model outputs to text and audio waveforms. The OmniUnifiedPostProcessor handles text cleaning while the Codec2WavProcessor converts codec tokens to audio. This modular design makes it reusable with any model producing similar output formats.