FFT Natural-Gradient Preconditioner

Integrate this into the optimizer immediately after backpropagation and before the parameter update. Start with convolutional kernels or spatially arranged model parameters whose last two dimensions are height and width; do not FFT across the channel or output-channel dimensions in the first experiment. For each eligible gradient tensor g with shape [out_channels, in_channels, H, W], precompute the real FFT frequency grid omega2 = wx2 + wy2 for the H x W grid and the fixed positive symbol lam = (alpha + beta*omega2)**s. Use the following pseudocode: g = parameter.grad; G = rfft2(g, dims=(-2,-1)); D = G / clamp(lam, min=eps); d = irfft2(D, s=(H,W), dims=(-2,-1)); parameter -= eta*d. Preserve the real-valued gradient after the inverse FFT. For an Adam-compatible variant, apply the FFT preconditioner to the raw gradient before computing Adam moments, but separately test applying it after Adam's bias-corrected direction; these are different algorithms. The mathematical computation is the Fourier diagonalization and modewise division. Estimate nothing for the fixed Sobolev baseline; for an adaptive extension estimate the per-frequency gradient power P_t(omega)=EMA(|FFT(g_t)(omega)|^2) and set lambda(omega)=clip((P_t(omega)+delta)^(-q/2), lambda_min, lambda_max) only after the fixed-symbol test. First run on CIFAR-10 with a small 6-layer CNN and compare SGD, SGD with momentum, AdamW, and this FFT-SGD at equal batch size and FLOPs. Sweep s in {0, 0.25, 0.5, 1} and choose beta from {0.01, 0.1, 1} after one validation split. Success means faster training loss decrease at equal optimizer steps, fewer exploding or oscillatory runs at large learning rates, or higher test accuracy at equal training compute; also report FFT overhead and whether the method improves convergence on a periodic-grid PDE task such as Burgers or Darcy flow.