KataGo is a high-performance Go AI engine implementing Monte Carlo Tree Search (MCTS) guided by deep neural networks, trained via AlphaZero-style self-play with numerous enhancements. The system functions as a command-line application with subcommands for GTP protocol (bot play), analysis, benchmarking, self-play training, model export, and distributed training coordination. It supports board sizes from 7×7 to 19×19, multiple rule sets, and human-style play imitation. The architecture is modular: a C++ core implements game logic (board, rules, history), MCTS search (search tree, node table, evaluation cache, time controls), and neural network inference through multiple backends (CUDA, ROCm, OpenCL, TensorRT, ONNX, Apple Metal, Eigen). A Python training pipeline handles self-play data generation, shuffling, model training (PyTorch), and export. The system relies on vendored GPU kernel libraries (NVIDIA CUTLASS, cuDNN frontend, AMD Composable Kernel FMHA) for efficient inference, and uses external utilities like TCLAP for CLI parsing and libzip for data I/O. Key data flows: self-play games produce training data (binary inputs, policy targets, value targets) stored as .npz files; these are shuffled, validated, and fed into a PyTorch training loop; trained models are exported and gated against previous versions before being deployed for further self-play. The distributed client coordinates model downloads, task execution, and upload of results across multiple workers. The codebase demonstrates strong engineering with extensive test coverage (search tests, backend reference tests, GTP protocol tests), reusable core utilities (thread-safe queues, parallel iteration, hashing, config parsing), and a clean separation between game logic, search, neural network backends, and training infrastructure.
This repository is a Python library that implements a comprehensive collection of reinforcement learning algorithms from Sutton and Barto's 'Reinforcement Learning: An Introduction'. It covers multi-armed bandits (epsilon-greedy, UCB, gradient bandit), dynamic programming (policy iteration, value iteration), Monte Carlo methods (first/every-visit, on/off-policy with importance sampling), temporal-difference learning (n-step SARSA, Q-learning, expected SARSA, double Q-learning, tree backup), planning (Dyna-Q, prioritized sweeping, trajectory sampling, MCTS), function approximation (gradient MC, semi-gradient TD, TD(λ), differential semi-gradient), and policy gradients (REINFORCE). The library is designed for educational use and experimentation, providing clean, modular implementations that closely follow the textbook. The architecture is organized into core data structures (ModelFree, MDP, policies, approximators) and solver functions grouped by category (model_based, model_free, approx, planning). Key entry points are the solver functions (e.g., alpha_mc, tdn, off_policy_mc, dynaq, mcts) which accept user-defined state/action spaces and transition functions, returning value functions and policies. Example scripts in the examples/ directory demonstrate usage on classic problems like blackjack, gridworld, mountain car, and random walk. Data flows are defined by the user providing a transition function that maps (state, action) to (next_state, reward, done); solvers then generate episodes and update internal tables or approximators. External dependencies are minimal: numpy for numerical operations, matplotlib for plotting (used in examples and UCTree visualization), tqdm for progress bars, and ruff/ty for development tooling. The library uses modern Python features (type hints, generics, protocols) and is packaged with hatchling. It requires Python 3.13+ and pins recent versions of its dependencies, indicating an up-to-date codebase.