# Geometric-Cycle Optimizer

- ID: 2720
- Canonical URL: https://synthcore.org/idea/2720/geometric-cycle-optimizer
- API JSON: https://synthcore.org/api/idea/2720.json
- API Markdown: https://synthcore.org/api/idea/2720.md
- Verification status: unverified
- Source: [arXiv:2608.24158](https://arxiv.org/abs/2608.24158)
- Category: dynamics
- Solves: stability, speedup, sample-efficiency
- ML areas: optimizer, scheduler, training-dynamics
- Math tags: dynamical-systems, stochastic-processes, geometry, control-theory
- Ratings: usefulness 6/10; difficulty 6/10; novelty 8/10

## Idea description

Augment an optimizer with two slowly and periodically modulated controls, such as learning rate and momentum or learning rate and gradient-noise scale. The optimizer state then traces a loop in control space; nonzero curvature can create a net parameter displacement that depends on loop orientation, even when the controls return to their initial values. Use curvature estimates to select loops that produce useful descent while penalizing loops with excessive dissipation.

## Mathematical statement

The paper's force-free elimination gives U = -G(x) x_dot, where x is a shape coordinate, U is translational velocity, and G(x) = zeta_Xx(x) / zeta_XX(x), with zeta the friction matrix. For a neural optimizer, let s be its stochastic internal state, theta = (theta_1, theta_2) two controls, L_theta the Markov generator of the optimizer state, rho_theta its stationary density, and j(s, theta) an observable parameter velocity. Define the geometric connection A_i(theta) = <j, L_theta^+ partial_i rho_theta>, where L_theta^+ is the zero-mode pseudoinverse and the brackets integrate over s. The curvature F_12 = partial_1 A_2 - partial_2 A_1 controls the geometric displacement of a slow closed loop. The excess cost is approximated by the quadratic form integral of g_ij theta_dot_i theta_dot_j, where g_ij is obtained from the pseudoinverse response of the stationary distribution.

## Key formulas

- $$U=-G(x)\dot{x},\qquad G(x)=\frac{\zeta_{Xx}(x)}{\zeta_{XX}(x)}$$
- $$A_i(\boldsymbol{\theta})=\left\langle j,L_{\boldsymbol{\theta}}^{+}\partial_{\theta_i}\rho_{\boldsymbol{\theta}}\right\rangle,\qquad F_{12}=\partial_{\theta_1}A_2-\partial_{\theta_2}A_1$$
- $$\Delta q_{\mathrm{geo}}\simeq\oint_C A_i\,d\theta_i=\iint_{S(C)}F_{12}(\boldsymbol{\theta})\,d\theta_1d\theta_2$$
- $$D_{\mathrm{ex}}\simeq\int_0^T g_{ij}(\boldsymbol{\theta})\dot{\theta}_i\dot{\theta}_j\,dt$$

## Implementation notes

1. Integration point: begin with Adam or SGD on a small MLP and expose two controls, for example theta_1 = log learning rate and theta_2 = momentum, or theta_2 = log injected gradient-noise variance. The optimizer state s_t consists of moments and weights; each minibatch transition is the stochastic dynamics. Drive the controls with theta_1(t) = mean_theta_1 + a cos(2 pi t / T) and theta_2(t) = mean_theta_2 + b sin(2 pi t / T + phi). 2. Pseudocode: sample a minibatch, compute gradient g_t, update the optimizer state and weights using the current controls, record a projected update j_t = negative dot product of g_t with a fixed validation-gradient direction, then advance the phase. At the end of each cycle, subtract the displacement produced by a constant-control baseline. Reverse the sine sign to reverse loop orientation. 3. Computed versus estimated: the paper supplies the connection, curvature, and metric mechanism, but a full generator pseudoinverse is unnecessary initially. Estimate A_i using finite differences from slow ramps at neighboring control points, then estimate F_12 with a centered numerical curl. Estimate dissipation from accumulated squared control velocity weighted by observed excess loss or gradient-noise energy. 4. First cheap experiment: train a two-layer MLP on MNIST with fixed minibatch count, comparing constant Adam, one-control sinusoidal Adam, and two-control elliptical cycles. Test periods T = 100, 300, and 1000, amplitudes a and b in {0.01, 0.03, 0.1}, and both orientations. The prediction is that the geometric displacement scales as a b F_12 for small amplitudes, changes sign under orientation reversal, and vanishes when either amplitude is zero. Excess cost should decrease approximately as 1/T at fixed loop geometry in the slow-driving regime; fast cycles should depart from this law and may destabilize training.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.