# Resonant Normal-Form Optimizer

- ID: 1283
- Canonical URL: https://synthcore.org/idea/1283/resonant-normal-form-optimizer
- API JSON: https://synthcore.org/api/idea/1283.json
- API Markdown: https://synthcore.org/api/idea/1283.md
- Verification status: unverified
- Source: [arXiv:2607.22464](https://arxiv.org/abs/2607.22464)
- Category: dynamics
- Solves: stability, speedup, accuracy
- ML areas: optimizer, training-dynamics, scheduler, rnn
- Math tags: dynamical-systems, bifurcations, control-theory, harmonic-analysis, numerical-analysis
- Ratings: usefulness 6/10; difficulty 6/10; novelty 8/10

## Idea description

Add a controlled periodic phase to an optimizer, then use a near-identity normal-form transform to remove rapidly oscillating gradient components instead of allowing them to perturb parameters directly. The optimizer follows averaged drift for non-resonant frequencies but explicitly preserves Fourier components near resonance, where they can create a secular update.

## Mathematical statement

Let \(w\in\mathbb{R}^d\) be network parameters, \(\varphi\in[0,2\pi)\) an optimizer phase with \(\dot\varphi=\omega\), and let \(\dot w=F(w,\varphi,t)\). Assume \(F(w,\varphi,t)=\sum_{k=1}^{N}t^{-k\alpha/2}F_k(w,\varphi)+R_N(w,\varphi,t)\), where \(\alpha>0\), each \(F_k\) is periodic in \(\varphi\), and \(R_N=O(t^{-(N+1)\alpha/2})\). Introduce the near-identity coordinate \(w=y+\sum_{k=1}^{N}t^{-k\alpha/2}U_k(y,\varphi)\). At order \(K\), the homological equation is \(\omega\partial_\varphi U_K=H_K-\langle H_K\rangle_\varphi\), where \(H_K\) is the order-\(K\) forcing after lower-order substitutions and \(\langle H_K\rangle_\varphi=(2\pi)^{-1}\int_0^{2\pi}H_K(y,\varphi)d\varphi\). The average cannot be removed by a periodic transformation and becomes effective slow drift. For Fourier mode \(m\), \(U_{K,m}=H_{K,m}/(im\omega)\); with an external gradient frequency \(\nu\), the denominator becomes \(i(m\omega-\nu)\), so small detuning identifies resonance.

## Key formulas

- $$\dot w=F(w,\varphi,t),\qquad \dot\varphi=\omega,\qquad F(w,\varphi,t)=\sum_{k=1}^{N}t^{-k\alpha/2}F_k(w,\varphi)+O\!\left(t^{-(N+1)\alpha/2}\right).$$
- $$w=y+\sum_{k=1}^{N}t^{-k\alpha/2}U_k(y,\varphi),\qquad \omega\,\partial_\varphi U_K=H_K-\langle H_K\rangle_\varphi.$$
- $$\langle H_K\rangle_\varphi=\frac{1}{2\pi}\int_0^{2\pi}H_K(y,\varphi)\,d\varphi,\qquad U_{K,m}=\frac{H_{K,m}}{im\omega}\quad(m\neq0).$$
- $$\left|m\omega-\nu\right|\leq\delta\ \Longrightarrow\ \text{retain mode }m;\qquad \left|m\omega-\nu\right|>\delta\ \Longrightarrow\ U_{K,m}=\frac{H_{K,m}}{i(m\omega-\nu)}.$$$

## Implementation notes

Integrate this at the optimizer update. At step \(n\), set \(\varphi_n=(\varphi_0+n\omega)\bmod 2\pi\), compute the minibatch gradient, and modulate either the learning rate or a low-rank update direction so the raw update is \(g_n=G(w_n,\varphi_n)\). Maintain exponential-memory Fourier estimates for \(|m|\leq M\): \(\widehat G_m\leftarrow(1-\beta)\widehat G_m+\beta g_n e^{-im\varphi_n}\). Estimate a dominant external gradient frequency \(\widehat\nu\) from the FFT or autocorrelation of recent scalar gradient projections. For each mode compute \(d_m=m\omega-\widehat\nu\). If \(|d_m|\leq\delta\), retain the mode; otherwise use the normal-form compensation \(U_m=\widehat G_m/(id_m)\), clipped at a maximum norm to avoid small-denominator explosions. Apply \(\Delta w_n=-\eta[\widehat G_0+\sum_{m\in\mathcal R}\widehat G_me^{im\varphi_n}]\), optionally reconstructing \(w=y+\sum_m t^{-\alpha/2}U_m\). The paper supplies the power-law expansion, homological equation, average-versus-oscillatory decomposition, and resonance denominator; Fourier coefficients, \(\widehat\nu\), and residuals are empirical estimates. First test a two-layer MLP on MNIST with SGD, cosine-periodic modulation, matched mean learning rate, and a sweep of \(\omega\). Measure correction norm, loss variance, and transformed residual \(r_n=\Delta w_n-\Delta y_n\). Prediction: away from resonance, residual RMS decays approximately as \(n^{-\alpha/2}\), while correction magnitude scales as \(1/|m\omega-\nu|\). A sharp peak should occur when \(|m\omega-\nu|\lesssim\delta\), and increasing truncation from \(N=1\) to \(N=2\) should improve the residual slope by roughly one additional factor of \(n^{-\alpha/2}\). Reject the mechanism if no detuning-dependent peak or power-law decay is observed.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.