LongCat-Flash-Omni is an application providing inference capabilities for a 560B parameter omni-modal model that processes text, audio, image, and video inputs to generate text and/or audio outputs. The system features a modular architecture with separate modality encoders (audio, vision), a unified data processor for multimodal inputs, and a post-processor for converting model outputs to usable formats. The main entry point is longcat_omni_demo.py which orchestrates the inference pipeline, leveraging SGLang for efficient model serving. The repository demonstrates strong engineering practices in its modular design, particularly in the audio feature processing and image transformation components.
MiniMind-O is a research codebase for training and evaluating a small-scale (0.1B parameter) omni-modal language model that processes text, audio, and image inputs and generates text or streaming speech. The architecture uses a Thinker–Talker dual-path design with frozen external encoders (SenseVoice, SigLIP2, Mimi codec, CAM++) and trainable projectors, plus a Multi-Token Prediction (MTP) head for audio codec tokens. The repository includes a complete training pipeline (SFT with DDP, mixed precision, checkpointing), a comprehensive evaluation CLI, a Gradio web demo, and a real-time Flask WebSocket demo with VAD and ASR for near-duplex interaction. Data is stored in parquet format with pre-encoded audio codes, images, and speaker embeddings. The project is primarily research-oriented, aiming to provide a transparent, reproducible baseline for omni-modal models, but also includes application-like components for interactive demos.