CurvLinOps

This library implements scipy LinearOperator s for deep learning matrices, such as

• the Hessian

• the Fisher/generalized Gauss-Newton (GGN)

Matrix-vector products are carried out in PyTorch, i.e. potentially on a GPU. The library supports defining these matrices not only on a mini-batch, but on data sets (looping over batches during a matvec operation).

You can plug these linear operators into scipy, while carrying out the heavy lifting (matrix-vector multiplies) in PyTorch on GPU. My favorite example for such a routine is eigsh that lets you compute a subset of eigenpairs.

Installation

pip install curvlinops-for-pytorch

Getting started

• How to use CurvLinOps

CurvLinOps

• Linear operators
  • Hessian
  • Generalized Gauss-Newton
  • Fisher (approximate)
  • Uncentered gradient covariance (empirical Fisher)
  • Jacobians
  • Inverses
  • Sub-matrices
• Spectral density approximation
  • lanczos_approximate_spectrum()
  • lanczos_approximate_log_spectrum()
  • LanczosApproximateSpectrumCached
• Trace approximation
  • HutchinsonTraceEstimator
  • HutchPPTraceEstimator
• Diagonal approximation
  • HutchinsonDiagonalEstimator
• Frobenius norm approximation
  • HutchinsonSquaredFrobeniusNormEstimator
• Experimental
  • ActivationHessianLinearOperator
• Code samples

Internals

• Internals
  • KFAC-related
    • loss_hessian_matrix_sqrt()