# Spectral Eigenmode-Sensitivity Damping

- ID: 2120
- Canonical URL: https://synthcore.org/idea/2120/spectral-eigenmode-sensitivity-damping
- API JSON: https://synthcore.org/api/idea/2120.json
- API Markdown: https://synthcore.org/api/idea/2120.md
- Verification status: unverified
- Source: [arXiv:2608.11309](https://arxiv.org/abs/2608.11309)
- Category: dynamics
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics, regularization
- Math tags: spectral-theory, linear-algebra, geometry, dynamical-systems
- Ratings: usefulness 6/10; difficulty 7/10; novelty 7/10

## Idea description

Track where the loss Hessian's eigenvectors are most sensitive to the current minibatch perturbation, rather than using only eigenvalues or a global learning-rate estimate. Apply extra damping only to spectral bands with high geometric response, allowing flat and well-separated curvature modes to retain a larger step size.

## Mathematical statement

The paper's fidelity susceptibility for an eigenstate |n> of a parameter-dependent Hamiltonian H(lambda) is chi_lambda^n = sum_{m != n} |<m|partial_lambda H|n>|^2/(E_n-E_m)^2, where E_n and |n> are the eigenvalue and normalized eigenvector of H, |m> are the other eigenvectors, and partial_lambda H is the perturbation operator. Its spectral response density is B_lambda(E) = sum_n delta(E-E_n) chi_lambda^n, where delta is the Dirac delta and E is the spectral coordinate. Adapt H to the symmetric minibatch Hessian H_t of the neural-network loss and partial_lambda H to a perturbation Delta H_t, for example the difference between minibatch Hessians on two independent batches. For Hessian eigenpairs (lambda_i,u_i), define chi_i = sum_{j != i} |u_j^T Delta H_t u_i|^2/(lambda_i-lambda_j)^2. Large chi_i means the curvature eigenvector rotates strongly under a small data or parameter perturbation; the denominator makes near-degenerate spectral regions especially sensitive. Estimate a bandwise density B_k = sum_{i in band k} chi_i and use it to set a band-dependent damping coefficient. The construction is heuristic for neural losses: unlike the quantum setting, the Hessian may be indefinite and only approximate eigenpairs are available, so use absolute or squared spectral gaps and clip extreme values.

## Key formulas

- $$\chi_{\lambda}^{n}=\sum_{m\ne n}\frac{\left|\langle m\mid \partial_{\lambda}\hat H\mid n\rangle\right|^{2}}{(E_n-E_m)^2}$$
- $$B_{\lambda}(E)=\sum_n\delta\left(E-E_n\right)\chi_{\lambda}^{n}$$
- $$\chi_i^{\mathrm{NN}}=\sum_{j\ne i}\frac{\left|u_j^{\mathsf T}\Delta H_tu_i\right|^2}{\left(|\lambda_i-\lambda_j|+\varepsilon\right)^2},\qquad B_k=\sum_{i:\lambda_i\in I_k}\chi_i^{\mathrm{NN}}$$
- $$\eta_i=\frac{\eta_0}{1+\alpha\,\widetilde\chi_{b(i)}},\qquad \Delta\theta_t=-\sum_i\eta_i\,(u_i^{\mathsf T}g_t)u_i$$

## Implementation notes

Integrate this into a second-order optimizer or a lightweight Hessian-aware wrapper around SGD or Adam. At training step t, use two minibatches A and B and estimate Hessian-vector products for H_A and H_B; set H_t=H_A and Delta H_t=H_B-H_A. Run Lanczos for r=32-64 iterations on H_t to obtain approximate eigenpairs (lambda_i,u_i) over the largest-magnitude curvature modes. Form the matrix M_ij=u_i^T Delta H_t u_j using Hessian-vector products, then compute chi_i from the displayed sum. Partition the estimated eigenvalues into 4-8 bands I_k, preferably logarithmic bands in |lambda|, and maintain an exponential moving average chi_bar_k=beta chi_bar_k+(1-beta)chi_k. Normalize each band response by its median or by the global mean, clip it to [0,C], and use eta_i=eta_0/(1+alpha chi_bar_{b(i)}) for the Lanczos subspace. For gradient components orthogonal to the subspace, retain the base optimizer update; alternatively use one global residual learning rate. Recompute the spectral estimate every 100-500 optimizer steps and stop using the correction if Lanczos residuals are too large. The paper supplies the susceptibility and density construction; the neural-network choices of Hessian perturbation, banding, clipping, and EMA must be tested empirically. First experiment: train a small 4-layer MLP or tiny ViT on CIFAR-10 with SGD at a learning rate near the baseline stability limit, comparing SGD, global Hessian damping, and this bandwise method at equal forward/backward FLOPs. Measure loss spikes, gradient norm variance, final validation accuracy, and wall-clock overhead. Success is fewer instability spikes and faster loss descent at the same stable peak learning rate, with no more than roughly 20% training overhead.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.