# Adaptive Harmonic Gradient Damping

- ID: 2537
- Canonical URL: https://synthcore.org/idea/2537/adaptive-harmonic-gradient-damping
- API JSON: https://synthcore.org/api/idea/2537.json
- API Markdown: https://synthcore.org/api/idea/2537.md
- Verification status: unverified
- Source: [arXiv:2608.18625](https://arxiv.org/abs/2608.18625)
- Category: dynamics
- Solves: stability, speedup
- ML areas: optimizer, training-dynamics, scheduler, rnn
- Math tags: dynamical-systems, control-theory, stochastic-processes, spectral-theory, numerical-analysis
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Treat the component of minibatch-gradient noise that is coherent across iterations as an unknown periodic disturbance, estimate its phase and frequency with a latent oscillator, and subtract an anti-phase update from the optimizer step. Unlike fixed momentum or a fixed low-pass filter, the oscillator estimates the disturbance frequency online and therefore does not require prior knowledge of the data period, sequence period, or model-specific time scale.

## Mathematical statement

The paper estimates an external disturbance from measured acceleration and known thrust, models the disturbance as periodic with unknown frequency, and applies active damping. The neural analogue uses the observed gradient residual r_t = g_t - gbar_t, where g_t is the minibatch gradient and gbar_t is an exponential mean with coefficient beta. A dominant oscillatory component is represented by z_t = [q_t, v_t]^T, where q_t is the estimated disturbance and v_t is its quadrature component. Its dynamics are z_{t+1} = A(omega_t) z_t + B r_t + w_t, where omega_t is angular frequency in radians per optimizer step, rho is a persistence factor, A(omega) is a damped rotation matrix, B injects the residual, and w_t is process noise. The optimizer applies Delta theta_t = -eta(g_t + kappa q_t), where theta_t are network parameters, eta is the base learning rate, and kappa is the damping gain. For a locally quadratic loss with curvature lambda, the augmented closed-loop system is stable when the spectral radius of its transition matrix M_cl is below one. The proposed energy signature is geometric decay of E_t = (q_t^2 + v_t^2)/2 + c||theta_t - theta*||^2 after frequency lock.

## Key formulas

- $$r_t=g_t-\bar g_t,\qquad \bar g_t=\beta\bar g_{t-1}+(1-\beta)g_t,$$
- $$z_{t+1}=A(\omega_t)z_t+B r_t+w_t,\qquad A(\omega)=\rho\begin{bmatrix}\cos\omega&\sin\omega\\-\sin\omega&\cos\omega\end{bmatrix},\quad B=\begin{bmatrix}1\\0\end{bmatrix},$$
- $$\Delta\theta_t=-\eta\left(g_t+\kappa q_t\right),\qquad \rho\!\left(M_{\mathrm{cl}}(\eta,\kappa,\lambda,\omega)\right)<1,$$
- $$E_{t+1}\leq\gamma E_t\quad (0<\gamma<1),\qquad E_t=\frac{1}{2}(q_t^2+v_t^2)+c\|\theta_t-\theta^*\|^2.$$

## Implementation notes

1. Integration point: insert the observer immediately after computing each minibatch gradient and before AdamW or SGD applies its parameter update. Maintain one oscillator per parameter block, layer, or low-dimensional gradient projection. Begin with one scalar oscillator driven by a projection of the residual gradient onto the normalized previous-gradient direction; this avoids storing a full extra parameter tensor. Compute g_t, update the exponential mean gbar_t, and form r_t. 2. Pseudocode: `g = grad(loss); mean = beta*mean + (1-beta)*g; r = g-mean; s = dot(r, direction); A = rho*rotation(omega); z_pred = A @ z + B*s; innovation = s - z_pred[0]; P = A @ P @ A.T + Q; K = P @ H.T @ inverse(H @ P @ H.T + R); z = z_pred + K*innovation; P = (I-K@H) @ P; omega = clip(omega + eps_omega*innovation*z[1], omega_min, omega_max); update = g + kappa*z[0]*direction; theta = optimizer_step(theta, update)`. Here H = [1, 0], Q is process-noise covariance, R is measurement-noise variance, and `direction` is the normalized recent gradient direction. For a full-vector variant, use a low-rank basis computed from the last m residual gradients. 3. The harmonic observer and anti-phase damping are transferred from the paper; rho, Q, R, kappa, and the frequency adaptation rate eps_omega are estimated or tuned empirically. Initialize rho = 0.995, beta = 0.99, and search frequencies from 2*pi/200 to 2*pi/2. 4. First cheap experiment: train an MLP and a two-layer GRU on MNIST and on a synthetic sequence dataset with a deliberately imposed period P = 20 in minibatch ordering or target forcing. Compare SGD, AdamW, and the observer optimizer at matched learning rates. Log the residual-gradient periodogram, estimated frequency, loss envelope, and a finite-difference estimate of the local update Jacobian spectral radius. The prediction is that omega_hat locks near 2*pi/P within approximately 100 steps, the targeted oscillation amplitude decays geometrically after lock, and the maximum stable learning rate increases relative to the uncontrolled optimizer. Sweep eta and kappa; the divergence boundary should agree within 20 percent with the numerically computed condition spectral_radius(M_cl) = 1. Excessive kappa should create a secondary oscillation or instability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.