Spectral Eigenmode-Sensitivity Damping

Integrate this into a second-order optimizer or a lightweight Hessian-aware wrapper around SGD or Adam. At training step t, use two minibatches A and B and estimate Hessian-vector products for H_A and H_B; set H_t=H_A and Delta H_t=H_B-H_A. Run Lanczos for r=32-64 iterations on H_t to obtain approximate eigenpairs (lambda_i,u_i) over the largest-magnitude curvature modes. Form the matrix M_ij=u_i^T Delta H_t u_j using Hessian-vector products, then compute chi_i from the displayed sum. Partition the estimated eigenvalues into 4-8 bands I_k, preferably logarithmic bands in |lambda|, and maintain an exponential moving average chi_bar_k=beta chi_bar_k+(1-beta)chi_k. Normalize each band response by its median or by the global mean, clip it to [0,C], and use eta_i=eta_0/(1+alpha chi_bar_{b(i)}) for the Lanczos subspace. For gradient components orthogonal to the subspace, retain the base optimizer update; alternatively use one global residual learning rate. Recompute the spectral estimate every 100-500 optimizer steps and stop using the correction if Lanczos residuals are too large. The paper supplies the susceptibility and density construction; the neural-network choices of Hessian perturbation, banding, clipping, and EMA must be tested empirically. First experiment: train a small 4-layer MLP or tiny ViT on CIFAR-10 with SGD at a learning rate near the baseline stability limit, comparing SGD, global Hessian damping, and this bandwise method at equal forward/backward FLOPs. Measure loss spikes, gradient norm variance, final validation accuracy, and wall-clock overhead. Success is fewer instability spikes and faster loss descent at the same stable peak learning rate, with no more than roughly 20% training overhead.