MiniMind is an educational open-source toolkit for training small language models entirely from scratch using native PyTorch. It provides a complete pipeline including custom BPE tokenizer training, pretraining, supervised fine-tuning (SFT), LoRA, RLHF (DPO), RLAIF (PPO/GRPO/CISPO), agentic reinforcement learning, and knowledge distillation. The system is designed to be accessible (low compute cost, ~3 RMB and 2 hours on a single GPU) and includes inference interfaces such as a CLI, Streamlit web UI, and an OpenAI-compatible API server. While primarily a software application for training and deploying LLMs, it also contains dataset preparation, evaluation benchmarks, and research experiments, giving it a hybrid character that leans toward application due to its focus on providing reusable training scripts and deployment tools.
OpenRLHF is a production-oriented distributed RLHF/RL training framework for large language models, not a research notebook collection. The codebase implements a full training stack: SFT, reward modeling, DPO/IPO/cDPO, and RL algorithms including PPO, REINFORCE++, GRPO, RLOO, Dr.GRPO, and FlashREINFORCE. Its architecture combines Ray for distributed actor orchestration and placement groups, vLLM for high-throughput rollout generation, DeepSpeed ZeRO for model training, and HuggingFace Transformers for model handling. The primary entry points are CLIs under openrlhf/cli, with train_ppo_ray.py as the main RL launcher; these accept a large set of hierarchically parsed dotted flags that control training, rollout, reward, vLLM, and DeepSpeed behavior. The repository is genuinely hybrid in spirit because it trains research-scale models and reproduces academic methods, but its dominant character is application/framework: it ships packaging, Dockerfiles, CLI entry points, unit tests, and versioned releases. The design emphasizes a token-in-token-out agent abstraction that unifies single-turn and multi-turn RLHF, plus sync and async training modes, a hybrid engine that colocates training and generation on shared GPUs, and VLM support. Distributed checkpointing, LoRA/QLoRA, packing, dynamic batch sizing, importance sampling corrections, and KL-estimator variants are all present in the code, not merely in the README. Craftsmanship is strong. The code separates concerns across cli/, trainer/, models/, datasets/, and utils/, and includes several genuinely reusable patterns such as a self-contained math answer grader, a flat-to-hierarchical argparse config converter, a typed RL experience dataclass, and distributed loss-normalization utilities. The test suite exercises loss estimators, trainer control flow, experience batching, and agent behavior with mocked heavy dependencies, which is a pragmatic way to test GPU-bound code on CPU. The main weaknesses are operational rather than architectural: dependencies are extremely tightly pinned (deepspeed==0.19.6, transformers==5.15.0, ray==2.55.0, vllm==0.29.0), and the code monkey-patches upstream internals and relies on version-specific behaviors. This makes upgrades brittle, and the CI pipeline builds wheels without running the test suite. Overall, this is a mature, actively maintained framework with high reusability for engineers building LLM alignment pipelines, though it demands GPU infrastructure and careful dependency alignment.
This repository is the official companion code for the book "Build a Reasoning Model (From Scratch)" by Sebastian Raschka (Manning, 2025). It is primarily a data_research codebase focused on building, evaluating, and training reasoning LLMs from scratch using the Qwen3 0.6B model as a base. The code covers inference-time scaling (temperature/top-p sampling, self-consistency), reinforcement learning from verifiable rewards (GRPO/RLVR), self-refinement, and knowledge distillation from larger teachers (DeepSeek-R1, Qwen3 235B). It includes a full custom Qwen3 implementation with KV cache, RoPE, and grouped-query attention, plus training scripts, notebooks, benchmark evaluations (MATH-500, MMLU), and Hugging Face export adapters. While educational in nature, it demonstrates production-quality engineering for research experiments. The repository is hybrid: it provides a reusable Python package for text generation and training, but its dominant character is data_research due to its notebook-driven, experimental, and benchmark-focused workflow.
LLaMA-Factory is a production-grade, unified framework for fine-tuning, evaluating, and serving 100+ large language and multimodal models. It provides a CLI (`llamafactory-cli`), a Gradio web UI (LLaMA Board), and an OpenAI-compatible API server. The architecture is monolithic but highly modular, built on Hugging Face Transformers/TRL/PEFT with optional integration of vLLM, SGLang, DeepSpeed, FSDP, Megatron Bridge, and hardware-specific backends (CUDA, ROCm, NPU, XPU). Training pipelines support full-parameter, LoRA, QLoRA, freeze-tuning, and alignment methods (DPO, PPO, KTO, etc.) with extensive data processing (multimodal, packing, streaming, template-based formatting). The codebase features a plugin system (v1) for kernels, quantization, distributed backends, and batching strategies, though it maintains a parallel legacy (v0) path. Reusable components include dataset processors, chat template hierarchy, multimodal plugin architecture, custom trainer callbacks, and batching strategy plugins. While the framework is actively maintained and well-tested, it suffers from some technical drift due to a dual-codebase structure, hardcoded model registries, version pinning, and fragile monkey-patching in distributed integrations.
Kiln is a full-stack AI development platform for building, evaluating, and optimizing LLM-based tasks and agents. It provides a desktop application (Tkinter tray + FastAPI server) with a Svelte web UI, plus a core Python library and a cloud server component. The system supports the complete AI workflow: project/task management, multi-provider LLM orchestration via LiteLLM, evaluation with multiple judge types (LLM-as-judge, code eval, exact match, pattern match, etc.), synthetic data generation, RAG pipelines (chunking, embedding, vector stores), fine-tuning, prompt optimization, and Git-based project versioning. It also includes a 'Kiln Pro' cloud service for advanced features like copilot-assisted spec refinement and batch generation. The architecture is a monorepo with uv workspace, featuring extensive test coverage (unit, integration, e2e) and CI/CD pipelines. While the primary character is an application, it incorporates significant data engineering patterns for evaluation pipelines, synthetic data generation, and statistical analysis.
ColossalAI is a comprehensive distributed training and inference framework for large-scale deep learning models, providing multiple parallelism strategies (data, tensor, pipeline, sequence, mixture-of-experts), ZeRO memory optimization, automatic parallelization via FX tracing, and high-performance custom CUDA/Triton kernels. The architecture is library-based with a plugin-style booster for orchestrating training and inference, a shardformer for model-level parallelism policies, and a legacy subsystem for older APIs. It includes application-level tools: ColossalChat for RLHF (PPO, DPO, GRPO), ColossalQA for retrieval-augmented question answering, ColossalEval for LLM evaluation with GPT judges, and Colossal-LLaMA for domain-specific LLM training. The codebase is actively maintained with recent commits, but has notable technical drift due to outdated dependency pins in some examples and a large legacy namespace that coexists with newer implementations.
This repository is a comprehensive family of GUI automation agents (Mobile-Agent series) developed by Tongyi Lab, Alibaba Group. It provides end-to-end solutions for automating interactions across mobile (Android), desktop (Windows/macOS/Linux), and web platforms using vision-language models. The system perceives GUI elements through screenshots, grounds natural language instructions to actionable commands (click, type, scroll), and executes tasks via platform-specific controllers. The architecture includes multiple agent versions (v1-v3.5, E, UI-S1, PC-Agent) with shared core modules for perception, action parsing, and controller interfaces, alongside training frameworks (VERL) and evaluation suites (Android World, MiniWoB, Mind2Web). Key entry points are version-specific run scripts (e.g., run_ma35.py) and API services for remote agent control. The system relies on external dependencies like ADB for Android, PyAutoGUI for desktop, Playwright for web, and VLMs (Qwen series) for reasoning.
This repository is a comprehensive educational resource focused on large language models (LLMs), reinforcement learning (RL), and related training algorithms. It contains over 100 original diagrams (SVG/PNG/PDF) covering topics such as LLM architecture, SFT, DPO, RLHF, PPO, GRPO, policy gradient, and reasoning optimization. The repo also includes a curated index of open-source LLMs and VLMs with links to papers, code, and configs. While not a traditional software application or data pipeline, it serves as a visual knowledge base for researchers and practitioners in AI alignment and model training. The only executable code is a few utility scripts for image processing; the main value lies in the meticulously crafted diagrams that explain complex concepts with clarity. The repo is actively maintained and frequently updated with new content.