SuperSuit is a Python library that provides a collection of lightweight, composable 'microwrappers' for preprocessing reinforcement learning environments. It supports both single-agent Gymnasium environments and multi-agent PettingZoo environments (AEC and Parallel APIs). The wrappers cover common transformations such as color reduction, frame stacking, resizing, dtype conversion, reward clipping, action scaling, and NaN handling. Additionally, it offers vectorization utilities to run multiple environment instances in parallel, including synchronous, asynchronous, and multiprocessing-based vector environments. The library is designed to be used as a preprocessing pipeline before feeding environments into RL algorithms. It is maintained primarily for compatibility with newer versions of PettingZoo and Gymnasium, with the stated intention of eventually merging relevant functionality into PettingZoo's own wrappers.
This repository is the official implementation of the SANA family of diffusion models for high-resolution image and video generation, developed by NVIDIA Research. It is primarily a research codebase, containing training and inference pipelines for multiple model variants: SANA (text-to-image), SANA-1.5, SANA-Sprint (few-step distillation), SANA-Video, SANA-Video 2.0, SANA-Streaming (long video), LongSANA, and SANA-WM (world model for camera-controlled video). The codebase also includes post-training with reinforcement learning (Sol-RL), ControlNet support, and various schedulers (flow matching, DPM-Solver, LCM, self-forcing). While the core purpose is research, the repository also provides deployable Gradio applications, conversion scripts to Hugging Face diffusers format, and integration with ComfyUI and SGLang, giving it a hybrid application character. The code is heavily optimized with custom Triton kernels for efficient attention (LiteMLA, Gated Delta Net) and includes distributed training utilities (FSDP, context parallelism). The repository is actively maintained with frequent commits and supports a wide range of model sizes (600M to 14B parameters).
This repository is the complete open-source hardware and software stack for the Roboto Origin bipedal robot, formerly known as Atom01. It is a hybrid monorepo covering mechanical design (CAD files, 3D printing, BOMs), electronics (PCB designs, firmware for a USB2CAN board), low-level ROS2 control (motor drivers, IMU, camera, inference), simulation and reinforcement learning training (roboparty_train with Isaac Sim/MuJoCo and PPO/AMP algorithms), navigation (LiDAR localization, UWB, path planning), and XR teleoperation. The repository is organized into multiple modules, each with its own build system (CMake, Zephyr, Python). The dominant character is an application (a robot system), but it includes a substantial data_research component in the training module where RL policies are trained using motion capture data. The codebase is well-structured, actively maintained, and provides a complete recipe for building and deploying a humanoid robot from scratch.
TorchFort is a library that bridges PyTorch with Fortran and C/C++ programs, primarily targeting HPC/scientific computing applications. It provides a C API that allows users to load PyTorch models (via TorchScript or built-in MLP), train them, and perform inference directly on user-provided arrays without data copying. The library supports supervised learning as well as reinforcement learning (RL) with several state-of-the-art algorithms: DDPG, TD3, SAC, and PPO. It includes comprehensive RL infrastructure such as replay buffers, rollout buffers, online running normalizers, and distributed training via MPI. Integration with Weights & Biases is available for experiment tracking. Architecturally, TorchFort consists of a thin C API layer, a set of C++ implementations using LibTorch, and Fortran bindings generated via ISO_C_BINDING. The core C++ code is organized into modules for model management, training, distributed communication, CUDA utilities, and RL subsystems. Configuration is externalized via YAML files, and the library is designed to be used as a static or shared library linked into user applications. Notable components include a prioritized replay buffer with sum-tree implementation (replay_buffer.h), an online running normalizer using Welford's algorithm with MPI synchronization (running_normalizer.h), and a tensor list abstraction for handling multiple typed arrays (tensor_list.h). The codebase demonstrates good engineering practices with extensive test coverage, error handling via return codes, and support for both CPU and GPU execution. While primarily an application/library for integrating PyTorch into HPC codes, the repository also contains research-quality RL algorithm implementations and data preprocessing components, giving it a hybrid character.
AgentFlow is a framework for building and training agentic LLM systems using reinforcement learning, specifically Flow-GRPO (Flow-based Group Refined Policy Optimization). It provides a modular architecture with four core roles—Planner, Executor, Verifier, and Generator—that coordinate via evolving memory and tool calls over multiple turns. The system supports both inference (via a Solver) and distributed RL training (via a Trainer/Client/Server architecture built on verl, vLLM, and Ray). It includes a multi-provider LLM engine abstraction (OpenAI, Anthropic, Gemini, DeepSeek, etc.), a tool system with metadata-driven discovery, and instrumentation for tracing and reward collection. The repository is heavily research-oriented, with benchmark datasets (AIME24, GPQA, GAIA, HotPotQA, etc.) and training pipelines for optimizing agent planners. While primarily an application framework, it has strong data/research components, making it a hybrid project with a focus on reproducible agentic reasoning experiments.
SDPO (Self-Distilled Policy Optimization) is a framework for training large language models using reinforcement learning algorithms, built upon the verl (Volcano Engine Reinforcement Learning) foundation. The system provides a modular architecture for implementing PPO, GRPO, SDPO, and other RL algorithms with distributed training capabilities via Ray. It integrates with inference engines like vLLM and SGLang, supports multiple model architectures (Llama, Qwen with Megatron optimization), and includes reward modeling for domains like mathematics and coding. The framework features extensible components for reward managers, tools, and agent interactions, with experimental capabilities for fully asynchronous training and multi-turn reasoning. While actively maintained, the system's complexity and extensive dependency ecosystem present moderate technical drift risks.
This repository provides an end-to-end reinforcement learning framework for Unitree legged robots (Go2, G1, H1, H2, A2, AS2, R1). It uses MuJoCo physics (via the custom `mjlab` package) to simulate locomotion and motion imitation tasks. The codebase includes training scripts (`scripts/train.py`) that run PPO (from `rsl_rl`) with configurable network architectures and multi-GPU support, exporting trained policies to ONNX format. The `deploy/` directory contains C++ applications that execute these policies on real hardware using ONNX Runtime and Unitree SDK2, orchestrated by a finite state machine. The system is split into two main tasks: velocity tracking (gait generation) and motion tracking (whole-body imitation from reference trajectories). Key architectural patterns include manager-based environment configuration, composable MDP terms (rewards, observations, terminations), curriculum learning, and a YAML-driven FSM for deployment. The repository is actively maintained, with recent commits and bundled third-party dependencies (MuJoCo binaries, ONNX Runtime) for cross-platform deployment.
Pixel-Reasoner implements a two-stage training framework for Vision-Language Models (VLMs) to enable pixel-space reasoning. The system first performs instruction tuning on synthesized reasoning traces to familiarize models with visual operations (zoom-in, select-frame), then applies curiosity-driven reinforcement learning to balance exploration between pixel-space and textual reasoning. Targeting visual reasoning benchmarks (V* bench, TallyQA-Complex, InfographicsVQA, MVBench), it provides pre-trained models and evaluation scripts. The architecture combines adapted OpenRLHF components with custom VLM modules, Ray-based distributed training, and specialized data processors for multimodal input. Key entry points include training scripts (single/multi-node) and evaluation scripts, with data flowing from Hugging Face datasets through tokenization/image processing to PPO optimization loops.
Visual Spatial Tuning (VST) is a research framework for enhancing Vision-Language Models (VLMs) with human-like visuospatial abilities, from perception to reasoning. It implements a two-stage pipeline: supervised fine-tuning (SFT) on 4.1M spatial perception samples (VST-P) followed by reinforcement learning (RL) on 135K reasoning samples (VST-R). The framework extends to Vision-Language-Action (VLA) models for robotic manipulation evaluated on the LIBERO benchmark. The repository contains custom model implementations (Qwen2.5VL, Qwen3VL) with sequence parallelism and FSDP support, data preprocessing scripts, evaluation benchmarks (VSIBench, 3D object detection on SUN RGB-D, VLM benchmarks), and a distributed RL training system (verl) with reward functions and advantage estimation. While primarily a research codebase, it includes production-quality training infrastructure and reusable data utilities. The code is actively maintained with recent commits, but has medium technical drift due to tight version pinning and reliance on an internal ByteDance framework (VeOmni) that is not publicly available, which may hinder full reproducibility.
This repository is the companion code for the second edition of Aurélien Géron's 'Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow'. It consists of 19 Jupyter notebooks that teach a wide range of machine learning and deep learning topics, including classification, regression, neural networks, CNNs, RNNs, NLP, reinforcement learning, and scalable deployment. The code is well-structured and annotated, designed for interactive learning. While not a production application, the notebooks contain several high-quality, reusable components such as custom Keras layers, training loops, and learning rate schedulers. The repository also includes Docker configurations and environment files to ensure reproducibility. However, the second edition is explicitly outdated, with libraries pinned to versions from 2021 (TensorFlow 2.6, Scikit-Learn 1.0), and the README directs users to the third edition and a PyTorch version. The data sources are standard benchmark datasets bundled as static files or downloaded from URLs that may change.
OpenAI Gym is a foundational Python library providing a standard API for Reinforcement Learning (RL) environments. It defines the core `Env` interface (`reset`, `step`, `render`) and includes a registry of environments spanning classic control (CartPole), physics simulations (MuJoCo, Box2D), and grid worlds. The architecture relies heavily on a composition-based wrapper pattern (`gym.wrappers`) to modify environment behavior without altering core logic, and includes a vectorization module (`gym.vector`) for parallel execution using shared memory. While the codebase demonstrates strong architectural craftsmanship in its abstraction layers and space definitions, it is in a maintenance phase with development migrated to Gymnasium. It relies on external compiled libraries (MuJoCo, Box2D, Pygame) which introduce platform-specific complexity and dependency drift.
This repository implements a Whole-Body Control (WBS) deployment system for Unitree G1 humanoid robots. It uses a finite state machine (FSM) architecture with multiple control states including Passive, FixedStand, Loco (locomotion), AMP, MJAMP, and WBC (Whole-Body Control). The system processes real-time sensor data from the robot's IMU and joint encoders, runs inference on ONNX models to generate joint position commands, and accepts gamepad input to switch between control states. The application is designed for real-time operation with proper scheduling and safety considerations.
LLaVA-NeXT is a large multimodal model (LMM) research codebase that extends the LLaVA family to support single-image, multi-image, video, and interleaved tasks. It provides training pipelines for supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (PPO, GRPO) with custom vision encoders (CLIP, SigLIP, EVA-CLIP, MLCD) and modular projector/resampler components. The repository also includes a full RL training framework (EasyR1/verl) for large-scale GRPO training using Ray, FSDP, and vLLM, as well as evaluation scripts for video and interleaved benchmarks. While primarily a research codebase with extensive training configurations and experimental scripts, it also contains serving infrastructure (Gradio, SGLang, controller-worker) for deployment. The code is well-structured with reusable vision encoder wrappers, data protocols, and RL algorithm implementations, but suffers from hardcoded internal paths, multiple overlapping RL implementations, and legacy training pipelines that increase maintenance burden.
This repository is a reinforcement learning library implementing algorithms from Sutton & Barto's 'Reinforcement Learning: An Introduction'. It provides a modular framework for model-free and model-based RL methods, including multi-armed bandits, Monte Carlo methods, temporal difference learning, planning algorithms (Dyna-Q, MCTS), and function approximation. The code is organized into core interfaces (rl/utils.py), model-free/model-based frameworks, solver modules, and examples demonstrating usage. While not production-ready, it serves as an educational resource and reference implementation of classic RL algorithms.
This repository is a high-frequency trading (HFT) research and development framework, primarily written in Java with a significant Python layer for reinforcement learning and data analysis. The actual system provides a modular architecture for backtesting and live execution of algorithmic trading strategies across multiple exchanges (Binance, Interactive Brokers, Kraken, MetaTrader). It supports market making algorithms (Avellaneda-Stoikov, ConstantSpread), factor investing, and a wide range of execution algorithms (TWAP, VWAP, Implementation Shortfall). Reinforcement learning integration is a key feature, with Gymnasium environments communicating via ZeroMQ to Java-based RL agents. The data pipeline ingests tick-level depth and trade data from Parquet/CSV files or live streams, generates OHLCV candles, and feeds into both traditional backtesting and RL training loops. While the framework is functionally rich and demonstrates advanced patterns (Disruptor ring buffers, object pools, time-series utilities), the codebase suffers from maintenance drift: many third-party libraries are vendored or outdated, hardcoded paths are scattered, and several modules contain commented-out or incomplete code. The project is a hybrid of a production-like trading engine and research code, with the dominant character being an application framework for algorithmic trading research.
Craftium is a voxel-based game engine (Luanti/Minetest fork) extended with reinforcement learning capabilities. The core system provides a moddable 3D world with block-based gameplay, Lua scripting API, client-server architecture, and procedural map generation. The craftium add-on supplies Python bindings, Gymnasium environment interfaces, and RL training scripts to enable machine learning research within the game world. The system supports both traditional gameplay and AI experimentation through environment-specific mods that define reward functions, observations, and termination conditions for agents.
This repository implements a real-time reinforcement learning training system for two classic arcade games: Robotron: 2084 and Tempest. The system uses MAME emulators with Lua scripts to extract game state and apply actions, communicating via TCP sockets to Python agents. The Python side includes multiple RL algorithms: a Rainbow DQN (distributional C51 with factored action heads) for Tempest, and both a DQN and a PPO (Set Transformer) variant for Robotron. Training is supported by prioritized experience replay, n-step returns, behavioral cloning from heuristic experts, and a live metrics dashboard with real-time charts and game preview. The repository also contains the original assembly source code for both games, included for reference but not actively used by the training pipeline. The codebase is functional and well-structured in parts, but suffers from duplication between the two game projects, legacy code in an '_old_engine' directory, and platform-specific dependencies. It is actively maintained with recent training configuration tweaks documented in notes.
FinRL-Meta is a comprehensive Python library and framework for applying deep reinforcement learning (DRL) to financial trading. It provides a modular, plug-and-play ecosystem consisting of market environments (stock, crypto, futures, forex, portfolio allocation, order execution optimization, market impact), data processors (Yahoo Finance, Alpaca, Binance, Tushare, etc.), and DRL agent wrappers (Stable-Baselines3, ElegantRL, RLlib). The library implements the full training-testing-trading pipeline, with CLI entry points (train.py, test.py, trade.py), backtesting engines (WonderTrader integration for futures), and paper trading capabilities (Alpaca). It is the market environment layer of the larger FinRL ecosystem and includes research-grade features such as market impact models, turbulence index, and ensemble strategies. The codebase is hybrid: while primarily an application/library, it contains numerous Jupyter notebooks, experimental sub-projects (order execution with qlib, high-frequency futures), and academic paper references. This reflects its dual role as both a reusable software framework and a research reproducibility platform. The architecture is well-structured with abstract base classes (data processors, impact models, environments), factory patterns, and consistent Gym/Gymnasium interfaces. However, the repository also carries technical debt from its research origins: hardcoded paths, mixed gym versions, deprecated library usage, and some incomplete or commented-out code. The most reusable components are the data processor base class (providing a common data pipeline) and the pluggable impact model hierarchy, both of which can be extracted with minimal modification for other financial RL or simulation projects.
This repository hosts the Hugging Face Deep Reinforcement Learning Course (v2.0), a free, open-source educational curriculum covering RL from fundamentals (Q-Learning, Policy Gradients) to advanced topics (PPO, A2C, Multi-Agent RL, Curiosity-Driven RL). It is primarily a collection of theoretical lessons (MDX files) and hands-on Jupyter notebooks designed to run on Google Colab. The notebooks train agents using libraries like Stable-Baselines3, CleanRL, ML-Agents, and Sample Factory, and push trained models to the Hugging Face Hub with evaluation videos and leaderboard comparison. While not a production application or data pipeline, the repo contains reusable training loops, evaluation helpers, and Hub packaging functions that can be extracted for other RL projects. The course is in low-maintenance mode, with some broken features (Unit 7 AI vs AI) and dependency drift (gym vs gymnasium), but the core content remains functional and well-structured.
This repository implements a hierarchical controller for quadruped locomotion that combines a high-level reinforcement learning policy with a low-level model predictive controller (MPC). The RL policy dynamically predicts weight parameters for the MPC, which computes optimal ground reaction forces and joint torques. The codebase includes Isaac Gym training environments for the Unitree A1, Go1, and Aliengo robots, along with a custom C++ MPC solver (OSQP-based) and a finite state machine for robust transient handling. While primarily a research project that bridges RL and MPC for legged robots, it also provides reusable robotics components such as state estimators, leg controllers, gait pattern generators, and foot swing trajectory planners. The work is based on the MIT Cheetah software and includes detailed development logs and configuration files for reproducible experiments.