# Auxiliary-energy neural optimizer

- ID: 1872
- Canonical URL: https://synthcore.org/idea/1872/auxiliary-energy-neural-optimizer
- API JSON: https://synthcore.org/api/idea/1872.json
- API Markdown: https://synthcore.org/api/idea/1872.md
- Verification status: unverified
- Source: [arXiv:2608.06099](https://arxiv.org/abs/2608.06099)
- Category: optimization
- Solves: stability, speedup
- ML areas: optimizer, training-dynamics, mlp
- Math tags: optimization, numerical-analysis, dynamical-systems
- Ratings: usefulness 5/10; difficulty 6/10; novelty 7/10

## Idea description

Replace the direct nonlinear loss step by a scalar-auxiliary-variable discretization of a gradient flow. The optimizer maintains an auxiliary value representing the square root of the nonlinear energy, so the coupled update has a discrete modified-energy decrease even when the step size is not restricted by the local curvature of the loss.

## Mathematical statement

The paper uses MSAV variables to linearize nonlinear energy terms while proving unconditional energy stability. Its displayed coefficient contains the characteristic normalization $1/(2\sqrt{E_1(\eta^n)+C_1})$, for example $\alpha_0=\frac{r^n}{\tau}+\frac{1}{2\sqrt{E_1(\eta^n)+C_1}}((f'(\eta^n),(\eta_0^{n+1}-\eta^n)/\tau)+\cdots)$, where $E_1$ is a nonlinear energy, $C_1>0$ makes the square root well-defined, $r^n$ is the auxiliary scalar, $\tau$ is the time step, $f'$ is the derivative of the nonlinear potential, and $(\cdot,\cdot)$ is an inner product. Adapt this to parameters $\theta\in\mathbb{R}^d$ by writing the training objective as $\mathcal{L}(\theta)=\frac12\theta^T H\theta+E(\theta)$, with symmetric positive-semidefinite $H$, nonlinear residual energy $E(\theta)\geq0$, and $C>0$. Define $b^n=\nabla E(\theta^n)/\sqrt{E(\theta^n)+C}$ and $r^n\approx\sqrt{E(\theta^n)+C}$. Use the SAV gradient-flow discretization $(\theta^{n+1}-\theta^n)/\tau=-P\mu^{n+1}$, $\mu^{n+1}=H\theta^{n+1}+r^{n+1}b^n$, and $(r^{n+1}-r^n)/\tau=\frac12(b^n)^T(\theta^{n+1}-\theta^n)/\tau$, where $P\succcurlyeq0$ is a diagonal or block-diagonal preconditioner. For exact arithmetic and fixed positive $P$, this gives the modified energy $\widetilde{\mathcal{E}}^n=\frac12(\theta^n)^TH\theta^n+(r^n)^2$ and the identity $\widetilde{\mathcal{E}}^{n+1}-\widetilde{\mathcal{E}}^n=-\tau(\mu^{n+1})^TP\mu^{n+1}\leq0$.

## Key formulas

- $$\alpha_0=\frac{r^n}{\tau}+\frac{1}{2\sqrt{E_1(\eta^n)+C_1}}\left((f'(\eta^n),\frac{\eta_0^{n+1}-\eta^n}{\tau})+(\mu_{\eta 0}^{n+1},\mathbf{u}^n\cdot\nabla\eta^n)-(\widetilde{\mathbf{u}}_0^{n+1},\mu_\eta^n\nabla\eta^n)+\cdots\right).$$
- $$b^n=\frac{\nabla E(\theta^n)}{\sqrt{E(\theta^n)+C}},\qquad \frac{\theta^{n+1}-\theta^n}{\tau}=-P\mu^{n+1},\qquad \mu^{n+1}=H\theta^{n+1}+r^{n+1}b^n,$$
- $$\frac{r^{n+1}-r^n}{\tau}=\frac12(b^n)^T\frac{\theta^{n+1}-\theta^n}{\tau},\qquad \widetilde{\mathcal{E}}^n=\frac12(\theta^n)^TH\theta^n+(r^n)^2,$$
- $$\widetilde{\mathcal{E}}^{n+1}-\widetilde{\mathcal{E}}^n=-\tau(\mu^{n+1})^TP\mu^{n+1}\leq0.$$

## Implementation notes

Integrate this at the optimizer-update stage, after the minibatch gradient is computed and before parameters are written back. Start with an MLP on MNIST or CIFAR-10, where $\theta$ is the flattened parameter vector conceptually but all operations should be implemented layerwise. Choose $H=\lambda I$ or a diagonal running curvature estimate, $P=\operatorname{diag}(p_i)$ with $p_i=1/(\sqrt{v_i}+\epsilon)$, and $E(\theta)=\mathcal{L}_{\mathrm{batch}}(\theta)$ after shifting it by a nonnegative running baseline if necessary. At step $n$, evaluate the minibatch loss $E_n$ and gradient $g_n$, set $b_n=g_n/\sqrt{E_n+C}$, and initialize $r_n=\sqrt{E_n+C}$ on the first step. Solve the scalar/vector coupled system without forming a dense matrix: substitute $\theta_{n+1}=\theta_n-\tau P(H\theta_{n+1}+r_{n+1}b_n)$, giving $\theta_{n+1}=A_n^{-1}(\theta_n-\tau P r_{n+1}b_n)$ with $A_n=I+\tau PH$. Compute two diagonal or conjugate-gradient solves, one for $A_n^{-1}\theta_n$ and one for $A_n^{-1}P b_n$, then determine the scalar $r_{n+1}$ from the auxiliary equation. Finally recompute the minibatch loss and optionally damp the step if the actual loss increases sharply. The paper supplies the auxiliary-variable normalization and energy-law structure; $H$, $P$, $C$, and minibatch estimates are engineering choices. First compare against AdamW and SGD on a 2-layer 512-unit MLP at equal forward/backward FLOPs, sweeping learning rates above the AdamW stability threshold. Measure loss explosions, gradient norms, final accuracy, and the modified-energy sequence; success is a substantially larger stable learning-rate range and smoother loss descent, with no worse accuracy at matched compute.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.