OpenRLHF is a production-oriented distributed RLHF/RL training framework for large language models, not a research notebook collection. The codebase implements a full training stack: SFT, reward modeling, DPO/IPO/cDPO, and RL algorithms including PPO, REINFORCE++, GRPO, RLOO, Dr.GRPO, and FlashREINFORCE. Its architecture combines Ray for distributed actor orchestration and placement groups, vLLM for high-throughput rollout generation, DeepSpeed ZeRO for model training, and HuggingFace Transformers for model handling. The primary entry points are CLIs under openrlhf/cli, with train_ppo_ray.py as the main RL launcher; these accept a large set of hierarchically parsed dotted flags that control training, rollout, reward, vLLM, and DeepSpeed behavior. The repository is genuinely hybrid in spirit because it trains research-scale models and reproduces academic methods, but its dominant character is application/framework: it ships packaging, Dockerfiles, CLI entry points, unit tests, and versioned releases. The design emphasizes a token-in-token-out agent abstraction that unifies single-turn and multi-turn RLHF, plus sync and async training modes, a hybrid engine that colocates training and generation on shared GPUs, and VLM support. Distributed checkpointing, LoRA/QLoRA, packing, dynamic batch sizing, importance sampling corrections, and KL-estimator variants are all present in the code, not merely in the README. Craftsmanship is strong. The code separates concerns across cli/, trainer/, models/, datasets/, and utils/, and includes several genuinely reusable patterns such as a self-contained math answer grader, a flat-to-hierarchical argparse config converter, a typed RL experience dataclass, and distributed loss-normalization utilities. The test suite exercises loss estimators, trainer control flow, experience batching, and agent behavior with mocked heavy dependencies, which is a pragmatic way to test GPU-bound code on CPU. The main weaknesses are operational rather than architectural: dependencies are extremely tightly pinned (deepspeed==0.19.6, transformers==5.15.0, ray==2.55.0, vllm==0.29.0), and the code monkey-patches upstream internals and relies on version-specific behaviors. This makes upgrades brittle, and the CI pipeline builds wheels without running the test suite. Overall, this is a mature, actively maintained framework with high reusability for engineers building LLM alignment pipelines, though it demands GPU infrastructure and careful dependency alignment.
This repository implements a complete reinforcement learning pipeline for whole-body motion tracking on humanoid robots, specifically Unitree G1 and XBot L7. The system uses a teacher-student architecture with Proximal Policy Optimization (PPO) to train policies that track reference motions from retargeted motion capture datasets. It features three training phases: a base training phase, an adaptation phase with domain randomization, and a finetuning phase. The environment is built on MJLab (a MuJoCo wrapper) and includes extensive domain randomization (e.g., motor parameters, body mass, center of mass, joint offsets, external forces, and a window load capacity curriculum for payload-aware training). The codebase is highly modular, with registered MDP components (observations, rewards, terminations, randomizations) and a flexible configuration system using Hydra. It also includes advanced features such as symmetry augmentation, differentiable forward kinematics, and a custom motion dataset pipeline with memory-mapped storage. While primarily a reinforcement learning training framework, it has strong data engineering aspects for processing motion datasets and generating training data for the student encoder. The repository is well-structured for research and sim-to-real transfer, but it is tightly coupled to specific robot assets and dependencies (torch, torchrl, mjlab) with pinned versions that may require adaptation for other environments.
This repository provides an end-to-end reinforcement learning framework for Unitree legged robots (Go2, G1, H1, H2, A2, AS2, R1). It uses MuJoCo physics (via the custom `mjlab` package) to simulate locomotion and motion imitation tasks. The codebase includes training scripts (`scripts/train.py`) that run PPO (from `rsl_rl`) with configurable network architectures and multi-GPU support, exporting trained policies to ONNX format. The `deploy/` directory contains C++ applications that execute these policies on real hardware using ONNX Runtime and Unitree SDK2, orchestrated by a finite state machine. The system is split into two main tasks: velocity tracking (gait generation) and motion tracking (whole-body imitation from reference trajectories). Key architectural patterns include manager-based environment configuration, composable MDP terms (rewards, observations, terminations), curriculum learning, and a YAML-driven FSM for deployment. The repository is actively maintained, with recent commits and bundled third-party dependencies (MuJoCo binaries, ONNX Runtime) for cross-platform deployment.
This repository implements a real-time reinforcement learning training system for two classic arcade games: Robotron: 2084 and Tempest. The system uses MAME emulators with Lua scripts to extract game state and apply actions, communicating via TCP sockets to Python agents. The Python side includes multiple RL algorithms: a Rainbow DQN (distributional C51 with factored action heads) for Tempest, and both a DQN and a PPO (Set Transformer) variant for Robotron. Training is supported by prioritized experience replay, n-step returns, behavioral cloning from heuristic experts, and a live metrics dashboard with real-time charts and game preview. The repository also contains the original assembly source code for both games, included for reference but not actively used by the training pipeline. The codebase is functional and well-structured in parts, but suffers from duplication between the two game projects, legacy code in an '_old_engine' directory, and platform-specific dependencies. It is actively maintained with recent training configuration tweaks documented in notes.
This repository is a comprehensive educational resource focused on large language models (LLMs), reinforcement learning (RL), and related training algorithms. It contains over 100 original diagrams (SVG/PNG/PDF) covering topics such as LLM architecture, SFT, DPO, RLHF, PPO, GRPO, policy gradient, and reasoning optimization. The repo also includes a curated index of open-source LLMs and VLMs with links to papers, code, and configs. While not a traditional software application or data pipeline, it serves as a visual knowledge base for researchers and practitioners in AI alignment and model training. The only executable code is a few utility scripts for image processing; the main value lies in the meticulously crafted diagrams that explain complex concepts with clarity. The repo is actively maintained and frequently updated with new content.