$ show stanfordmlgroup/ngboost
NGBoost is a Python library for probabilistic gradient boosting that produces full predictive distributions rather than point estimates. It implements the Natural Gradient Boosting algorithm described in the research paper (arXiv 1910.03225). The library provides a scikit-learn-compatible API with NGBRegressor, NGBClassifier, and NGBSurvival classes, supporting a wide range of distributions (Normal, Bernoulli, Poisson, Gamma, Weibull, Beta, MultivariateNormal, etc.) and scoring rules (LogScore, CRPScore). A standout feature is the SymPy-based distribution factory (sympy_utils) that automatically derives gradients and Fisher Information from symbolic expressions, enabling easy creation of custom distributions. The architecture is modular: a core boosting loop (ngboost.py) works with pluggable base learners (learners.py), distributions (distns/), and scores (scores.py), combined via a manifold pattern (manifold.py). The library also includes survival analysis with censored scores, partial_fit for incremental learning, and backward-compatible model loading. While primarily a software library, it has strong research roots with experimental scripts, figure generation, and extensive Jupyter notebook examples. The codebase is well-structured and tested, though some deprecated sklearn usage and duplicated documentation content indicate moderate technical drift.
The LLM identified 3 files worth highlighting for reuse:
Factory module that auto-derives score, gradient, and Fisher Information from symbolic SymPy expressions. Enables users to define new distributions by providing a symbolic PDF or score expression, and the module automatically generates the necessary methods for natural gradient boosting. This is a highly reusable pattern for any probabilistic ML library that needs to support custom distributions without manual calculus.
Provides the `manifold` function that dynamically composes a Score class and a Distribution class into a single combined class with methods for score, gradient, and metric. This clean composition pattern decouples scoring rules from distribution parameterizations, allowing any compatible pair to be mixed. The resulting class is used as the core optimization target in the boosting loop.
Defines the abstract Score base class and concrete implementations (LogScore, CRPScore) for probabilistic model evaluation. Each score computes the loss, gradient, and Fisher Information (Riemannian metric) for natural gradient descent. The design supports both closed-form and Monte Carlo metric computation, and is extensible for custom scoring rules. This module is the mathematical backbone of the natural gradient optimization.