# Triangular Slow-Mode Optimizer

- ID: 3047
- Canonical URL: https://synthcore.org/idea/3047/triangular-slow-mode-optimizer
- API JSON: https://synthcore.org/api/idea/3047.json
- API Markdown: https://synthcore.org/api/idea/3047.md
- Verification status: unverified
- Source: [arXiv:2609.02351](https://arxiv.org/abs/2609.02351)
- Category: dynamics
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics, initialization
- Math tags: dynamical-systems, renormalization, control-theory, spectral-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Split the optimizer state into a passive descent sector and an explicitly nonequilibrium current-like sector, then constrain the local update Jacobian to be block triangular. The active sector may perturb the passive update, but passive perturbations cannot feed back into the active sector at first order, making its decay rate measurable and preventing uncontrolled oscillatory feedback. A second design objective tunes the two decay rates to reproduce the paper's slow-mode splitting and uses the measured crossover as a schedule for disabling the active sector.

## Mathematical statement

Let a=(a_eq,a_c,a_J)^T denote amplitudes of passive/equilibrium, passive-conserved, and active-current perturbations, and let ell be logarithmic scale. The paper obtains partial_ell a=M_act a with M_act block triangular: the last row has zero entries in the first two columns, so the current mode does not receive linear forcing from the passive modes. The corresponding eigenvalues are scaling exponents y_Theta and y_J, with y_Theta<0, y_J<0, and y_J-y_Theta=-eta, where eta is the anomalous dimension. Transfer this to optimizer coordinates x and z: x is the ordinary parameter or momentum state and z is an active correction state. Estimate the local update Jacobian J_opt and penalize its lower-left block J_zx, while allowing an upper-right block J_xz.

## Key formulas

- $$\partial_{\ell}\mathbf{a}=\mathsf{M}_{\rm act}\mathbf{a},\qquad \mathsf{M}_{\rm act}=\begin{pmatrix}* & * & *\\ * & * & *\\ 0 & 0 & c_a\end{pmatrix}.$$
- $$y_J-y_{\Theta_1}=-\eta.$$
- $$\frac{|a_J(\ell)|}{|a_{\Theta_1}(\ell)|}\approx\frac{|a_J(0)|}{|a_{\Theta_1}(0)|}e^{-\eta\ell},\qquad \ell_{\times}\approx\frac{1}{\eta}\log\frac{|a_J(0)|}{|a_{\Theta_1}(0)|}.$$
- $$\mathcal{L}_{\rm tri}=\lambda_{\rm tri}\left\|\widehat{J}_{zx}\right\|_F^2.$$

## Implementation notes

1. Integration point: implement AdamW or SGD with an additional optimizer state z_t having the same shape as the model parameters. Let x_t be the ordinary parameter update state. Use x_{t+1}=x_t-eta_g g_t+alpha z_t and z_{t+1}=rho z_t-eta_a h_t, where g_t is the current gradient and h_t is an exponential moving average of gradients. Detach h_t from the computation of z_t so that z influences x, while x does not feed back into the active-state evolution rule. This realizes an approximately triangular state transition. 2. Pseudocode: compute g=grad(L,x); update h=beta*h+(1-beta)*g; update z=rho*z-eta_a*h.detach(); update x=x-eta_g*g+alpha*z. Every K steps, estimate the sensitivity of z_{t+1} to x_t using a random-vector finite difference or a Jacobian-vector product, and add lambda_tri times its squared norm to a hyperparameter objective. Track A_Theta=norm(g) and A_J=norm(z), after projecting z orthogonally to g. 3. Taken from the paper: the triangular coupling structure and the prediction that the mode-rate gap is controlled by an anomalous exponent. Estimated empirically: eta, decay rates from regressions of log amplitudes against log training time, and the Jacobian blocks. 4. First experiment: train a two-layer MLP on MNIST with SGD, AdamW, and this optimizer using identical learning-rate grids. Initialize z so A_J/A_Theta=100 and sweep alpha. The falsifiable signature is a crossover satisfying log(t_cross) approximately equal to eta_inverse times log(A_J/A_Theta), with A_J/A_Theta decaying as t^{-eta}. Also test whether the measured lower-left Jacobian norm divided by the upper-right norm remains below 0.2 and whether the divergence threshold in alpha is higher than for an unconstrained bidirectional two-state optimizer.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.