$ show jingyaogong/minimind-o
MiniMind-O is a research codebase for training and evaluating a small-scale (0.1B parameter) omni-modal language model that processes text, audio, and image inputs and generates text or streaming speech. The architecture uses a Thinker–Talker dual-path design with frozen external encoders (SenseVoice, SigLIP2, Mimi codec, CAM++) and trainable projectors, plus a Multi-Token Prediction (MTP) head for audio codec tokens. The repository includes a complete training pipeline (SFT with DDP, mixed precision, checkpointing), a comprehensive evaluation CLI, a Gradio web demo, and a real-time Flask WebSocket demo with VAD and ASR for near-duplex interaction. Data is stored in parquet format with pre-encoded audio codes, images, and speaker embeddings. The project is primarily research-oriented, aiming to provide a transparent, reproducible baseline for omni-modal models, but also includes application-like components for interactive demos.
The LLM identified 4 files worth highlighting for reuse:
Core Omni model implementing the Thinker–Talker architecture with a bridge layer connecting the base LLM to a Talker module. The Talker uses a shared base embedding with per-layer low-rank adapters for MTP of 8-layer Mimi audio codes. Supports streaming generation with caching and voice cloning via reference audio codes and speaker embeddings. This is a reusable pattern for building multi-modal LLMs with separate reasoning and generation paths.
Dataset class that loads parquet files containing conversations, audio codes, images, and speaker embeddings. Handles prompt creation, audio augmentation (speed, noise, reverb, SpecAugment), image encoding via SigLIP2, and label generation for both text and audio codec tokens. Implements scheduled sampling for audio codec prediction. This is a well-structured, parameterized data pipeline that can be adapted for other omni-modal training tasks.
Flask-based real-time WebSocket demo for interactive voice chat with the model. Includes VAD (Silero VAD), ASR (SenseVoice), camera input, voice cloning modal, and streaming audio output via Mimi codec. The RealtimeSession class manages the VAD+ASR loop and barge-in. This is a reusable pattern for building near-duplex voice interfaces with local models.
Training script for supervised fine-tuning of the Omni model. Supports configurable training modes (all, audio_proj, vision_proj), DDP, gradient accumulation, mixed precision (bf16/fp16), learning rate scheduling, and checkpoint/resume. Includes evaluation during training and logging. This is a reusable training pipeline that can be adapted for other multi-modal models.