$ account jingyaogong
MiniMind is an educational open-source toolkit for training small language models entirely from scratch using native PyTorch. It provides a complete pipeline including custom BPE tokenizer training, pretraining, supervised fine-tuning (SFT), LoRA, RLHF (DPO), RLAIF (PPO/GRPO/CISPO), agentic reinforcement learning, and knowledge distillation. The system is designed to be accessible (low compute cost, ~3 RMB and 2 hours on a single GPU) and includes inference interfaces such as a CLI, Streamlit web UI, and an OpenAI-compatible API server. While primarily a software application for training and deploying LLMs, it also contains dataset preparation, evaluation benchmarks, and research experiments, giving it a hybrid character that leans toward application due to its focus on providing reusable training scripts and deployment tools.
This repository provides a minimal, from-scratch implementation of a Vision-Language Model (VLM) called MiniMind-V, designed for educational purposes and low-resource training. It ingests the ALLaVA-4V dataset (Parquet format with image bytes and JSON conversations) for pretraining (1.27M caption pairs) and supervised fine-tuning (2.9M mixed instruct/caption/text). The architecture follows a LLaVA-1.5 style: a frozen SigLIP2 visual encoder (256×256, 64 patch tokens) projects via a 2-layer MLP into a MiniMind LLM (with optional Mixture-of-Experts). Training freezes the visual encoder and trains only the projector and selected LLM layers, achieving ~3 RMB / 2 hours on a single RTX 3090. The codebase includes training scripts (pretrain and SFT), evaluation, a Gradio web demo, and model conversion to HuggingFace Transformers format. While primarily a research/educational project, it demonstrates a clean, modular VLM pipeline that can be adapted for other multimodal tasks.
MiniMind-O is a research codebase for training and evaluating a small-scale (0.1B parameter) omni-modal language model that processes text, audio, and image inputs and generates text or streaming speech. The architecture uses a Thinker–Talker dual-path design with frozen external encoders (SenseVoice, SigLIP2, Mimi codec, CAM++) and trainable projectors, plus a Multi-Token Prediction (MTP) head for audio codec tokens. The repository includes a complete training pipeline (SFT with DDP, mixed precision, checkpointing), a comprehensive evaluation CLI, a Gradio web demo, and a real-time Flask WebSocket demo with VAD and ASR for near-duplex interaction. Data is stored in parquet format with pre-encoded audio codes, images, and speaker embeddings. The project is primarily research-oriented, aiming to provide a transparent, reproducible baseline for omni-modal models, but also includes application-like components for interactive demos.