lixilinx/psgd_torch - thehardwarestore.ai

── AI DEEP ANALYSIS ──

reviewed by: nvidia/nemotron-3-super-120b-a12b

This repository provides a PyTorch implementation of Preconditioned Stochastic Gradient Descent (PSGD), a general-purpose second-order optimizer. It implements various preconditioner structures including Kronecker product, low-rank approximation, dense matrix, affine group, and X-matrix preconditioners. The code offers both Newton-type and whitening preconditioners with multiple update rules for the preconditioner matrices. Key features include support for different geometries (Q0.5EQ1.5, EQ, QEP, etc.), momentum handling, gradient clipping, and numerical stability mechanisms. The library includes optimizers suitable for various deep learning applications as demonstrated in the provided examples (vision transformers, GPT2, LSTMs, CNNs, etc.).

── METRICS ──

RATING9.0/10

QUALITY8.5/10

MAINTENANCEactive

DRIFTlow

CONTRIBUTORS1

VELOCITY0.79

OPEN ISSUES0

CLOSED ISSUES4

LAST COMMIT4 months ago

── TAGS ──

#optimizer #pytorch #preconditioner #second-order-optimization #kronecker-product #low-rank-approximation #machine-learning #deep-learning #lie-groups #hessian-vector-product #optimization-algorithms #stochastic-gradient-descent #affine-group #kronecker-factored-approximation #newton-schulz

── GITHUB TOPICS ──

lie-groups pytorch second-order-optimization hessian-vector-product deep-learning optimization-algorithms preconditioner stochastic-gradient-descent affine-group low-rank-approximation kronecker-factored-approximation newton-schulz

── FILE BREAKDOWN ──

Files by Type (count)

Python

27

Other (.svg)

4

Markdown

1

Other (.pdf)

1

Files by Type (size)

Other (.svg)

1MB

Python

367KB

Other (.pdf)

178KB

Markdown

23KB

── KEY FINDINGS: REUSABLE CODE ──

The LLM identified 1 files worth highlighting for reuse:

Contains the KronWhiten and KronNewton classes implementing sophisticated Kronecker product preconditioners with multiple update rules (Q0.5EQ1.5, EQ, QEQ, QUAD, etc.). These components are highly reusable as they depend only on torch and opt_einsum, demonstrate non-obvious mathematical skill in implementing advanced optimization techniques, and provide state-of-the-art preconditioning for deep learning models.

file: psgd.py ↗ | tags: #optimizer-pattern #kronecker-product-preconditioner #second-order-optimization #pytorch-optimizer

── HISTORY ──

Last analyzed: 2026-08-11 10:00:03.481313

Next check: 2026-08-18 10:00:03.325845

$ show lixilinx/psgd_torch