The repository provides a comprehensive ecosystem for the MiniCPM-V and MiniCPM-o series of multimodal large language models (MLLMs). It enables efficient on-device inference and real-time full-duplex multimodal interaction (vision + audio + text) with low latency. The core contributions include the OmniLMM architecture with intra-ViT early compression for visual encoding, mixed 4x/16x visual token compression, and support for streaming audio/video inputs. The system includes evaluation frameworks for multimodal benchmarks (MMBench, Video-MME, OmniDocBench), fine-tuning utilities (LoRA and full model), and deployment tools for various platforms (iOS, Android, HarmonyOS, web) and inference backends (vLLM, SGLang, llama.cpp, Ollama). Key entry points are the model inference code in omnilmm/model/, evaluation scripts in eval_mm/vlmevalkit/, fine-tuning in finetune/, and web demos in web_demos/ demonstrating real-time chat with multimodal inputs.
LMFlow is a comprehensive toolkit for fine-tuning, aligning, and deploying large foundation models. It provides a modular library with support for various efficient fine-tuning methods (LoRA, QLoRA, LISA), alignment techniques (DPO, RAFT), and inference backends (vLLM, SGLang). The architecture separates concerns into models, pipelines, optimizers, and utilities, with factory patterns for extensibility. Key features include conversation templating for multiple model families, custom optimizers, and distributed training integration. The toolkit is actively maintained with recent commits and addresses current research in model alignment and efficiency.
FireRedASR is an open-source industrial-grade automatic speech recognition (ASR) system supporting Mandarin, Chinese dialects, and English. It provides two model variants: FireRedASR-AED (Attention-based Encoder-Decoder with Conformer encoder and Transformer decoder) and FireRedASR-LLM (Encoder-Adapter-LLM framework leveraging Qwen2-7B). The codebase includes a CLI tool (`speech2text.py`), a Python API (`FireRedAsr`), feature extraction (fbank with CMVN), tokenization (mixed Chinese/English SentencePiece), and utilities for WER computation. It also includes a complete Triton Inference Server deployment with TensorRT-LLM acceleration for the AED model, with Docker support and benchmarking scripts. The architecture is modular: data preprocessing, model definitions, tokenizers, and inference orchestration are cleanly separated, though the model components are tightly coupled to the specific architectures. The repository is actively maintained with recent commits and demonstrates strong engineering craftsmanship in both the core ASR implementation and the production-grade deployment stack.
LongCat-Flash-Omni is an application providing inference capabilities for a 560B parameter omni-modal model that processes text, audio, image, and video inputs to generate text and/or audio outputs. The system features a modular architecture with separate modality encoders (audio, vision), a unified data processor for multimodal inputs, and a post-processor for converting model outputs to usable formats. The main entry point is longcat_omni_demo.py which orchestrates the inference pipeline, leveraging SGLang for efficient model serving. The repository demonstrates strong engineering practices in its modular design, particularly in the audio feature processing and image transformation components.
Higgs Audio v2 is an end-to-end multimodal LLM for expressive text-to-speech and audio generation from Boson AI. This repository contains the full v2 codebase: a Llama-3.2-3B backbone augmented with a DualFFN audio adapter, a unified 25 fps audio tokenizer (semantic + acoustic RVQ codes), ChatML-based data collators, and a serving engine with CUDA graph capture and multi-bucket static KV caches. The model supports zero-shot voice cloning, multi-speaker dialogue generation, and emergent capabilities like humming and background music. The README explicitly states that Higgs Audio v3 is released separately and this repo is only needed for v2/v2.5 reference. The code is organized as a Python package (boson_multimodal) with clear separation between model definition, audio processing, data collation, and serving. The serving engine (HiggsAudioServeEngine) provides both synchronous and async streaming generation. The generation loop is a custom implementation on top of HuggingFace's GenerationMixin that interleaves text token sampling with multi-codebook audio token sampling using a delay pattern. Notable engineering: DualFFN architecture (separate audio FFN paths in selected LLM layers), CUDA graph capture for decode, static KV cache buckets of different sizes, repetition-aware sampling (RAS) for audio tokens, and partially-frozen embedding/linear modules for efficient fine-tuning of new audio tokens. Caveats: the codebase pins transformers to <4.47.0 and relies on internal transformers APIs (LlamaDecoderLayer, LLAMA_ATTENTION_CLASSES), making it fragile against library upgrades. Some files contain broken imports (ac.py references a missing binary module; core_vq.py imports an external xcodec package) but these appear to be dead code paths. The README redirects users to v3, so this repo is primarily useful as v2 reference or for reproducing v2 results.
Monkey is a research codebase for multimodal large language models (MLLMs) focused on high-resolution image understanding and document-level text comprehension. It provides model definitions (Qwen-based Monkey/TextMonkey and InternVL-based Mini-Monkey), training scripts with DeepSpeed, Gradio demos, and an extensive evaluation harness covering dozens of VQA, captioning, and document understanding benchmarks. The repo also includes a data generation pipeline using BLIP2, GRiT, SAM, PaddleOCR, and ChatGPT to create detailed image captions and instruction-following data. Architecturally, it is a collection of Python scripts and modules rather than a deployable service, with heavy reuse of external libraries (transformers, flash-attn, detectron2) and research-grade code that exhibits duplication, hardcoded paths, and mixed model families. While functionally complete for its intended research purpose, the codebase lacks modularity and polish for production reuse, but contains several high-quality, self-contained components (e.g., token compression via PerceiverResampler, window attention, VQA evaluation metrics) that are extractable for other MLLM projects.