Fourier-Diagonalized Natural Gradients and Sobolev Mirror Descent

What the math gives to ML

The paper identifies a concrete class of natural-gradient metrics that become elementwise diagonal after a Fourier transform: a positive translation-invariant operator with symbol \(\lambda(\omega)\) turns the inverse-metric step into division by \(\lambda(\omega)\) at each frequency. This is directly transferable as an FFT preconditioner for spatial neural-network gradients, especially convolutional and operator-learning models whose parameters or activations live on periodic grids. A particularly usable default is the Sobolev/Bessel spectral filter, which suppresses high-frequency updates according to a tunable smoothness exponent; the safest first test is to apply it to SGD gradients while enforcing a positive symbol floor and comparing loss descent and stability against SGD and Adam.