# Symphony Constrained-Velocity Optimizer

- ID: 2051
- Canonical URL: https://synthcore.org/idea/2051/symphony-constrained-velocity-optimizer
- API JSON: https://synthcore.org/api/idea/2051.json
- API Markdown: https://synthcore.org/api/idea/2051.md
- Verification status: unverified
- Source: [arXiv:2608.09525](https://arxiv.org/abs/2608.09525)
- Category: dynamics
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: control-theory, dynamical-systems, optimization, stability
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace direct parameter updates with a hierarchical controller. An upper loop converts the minibatch gradient into a bounded desired parameter velocity, while a lower loop drives the actual velocity toward that reference through feedback and feedforward compensation. This should suppress minibatch-induced velocity spikes, make the maximum parameter displacement explicit, and preserve stable behavior when gradient estimates or curvature models are inaccurate.

## Mathematical statement

The source model has the controlled second-order form $m\dot v=f_{ex}+f_u-f_h(z)-\tilde f_r(v,\dot v)+\Delta$, where $z$ is position, $v$ velocity, $m>0$ mass, $f_u$ control, $f_h$ restoring force, $\tilde f_r$ damping, and $\Delta$ uncertainty. Transfer this structure to parameters $\theta$: $\dot\theta=v$ and $M\dot v=u$, with diagonal positive mass matrix $M$. The upper loop computes $v^\star=-P\,\widehat g$ and enforces a hard componentwise speed bound using $v^\star\leftarrow V_{\max}\tanh(-P\widehat g/V_{\max})$, where $\widehat g$ is the minibatch gradient, $P\succ0$ is a diagonal preconditioner, and $V_{\max}>0$ is the maximum parameter velocity. The lower loop uses $u=M\dot v^\star-K(v-v^\star)$, with diagonal $K\succ0$. For tracking error $e=v-v^\star$, the ideal continuous-time dynamics are $\dot e=-M^{-1}Ke$, giving exponential decay rate at least $\lambda_{\min}(M^{-1}K)$. With explicit Euler step $h$, tracking is linearly stable when $\rho(I-hM^{-1}K)<1$, equivalently $0<h<2/\lambda_{\max}(M^{-1}K)$ for symmetric positive-definite diagonal gains. Near a local minimum with Hessian eigenvalues $\lambda_i>0$ and unsaturated reference velocity, the position dynamics reduce to $\dot\theta=-P H(\theta-\theta^\star)$ and decay at rates given by eigenvalues of $PH$.

## Key formulas

- $$m\dot v=f_{ex}+f_u-f_h(z)-\tilde f_r(v,\dot v)+\Delta$$
- $$\dot\theta=v,\qquad M\dot v=u,\qquad v^\star=V_{\max}\tanh\left(-P\widehat g/V_{\max}\right)$$
- $$u=M\dot v^\star-K(v-v^\star),\qquad \dot e=-M^{-1}Ke,\quad e=v-v^\star$$
- $$\rho\left(I-hM^{-1}K\right)<1\quad\Longleftrightarrow\quad 0<h<\frac{2}{\lambda_{\max}(M^{-1}K)}$$

## Implementation notes

Integrate this at the optimizer level, without changing the neural-network forward pass or loss. Maintain parameter velocity v and optionally an exponential-moving-average gradient g. At each minibatch: (1) compute the ordinary gradient g_hat; (2) update the gradient estimate g_hat <- beta*g_hat_old+(1-beta)*g_batch; (3) form the constrained reference velocity v_star = Vmax*tanh(-P*g_hat/Vmax), elementwise; (4) estimate its derivative with finite differences, dv_star = (v_star-v_star_old)/h, or omit the feedforward term during the first ablation; (5) calculate acceleration a = dv_star - M^{-1}K*(v-v_star); (6) update v <- v+h*a; (7) update theta <- theta+h*v. Use diagonal M, K, and P, initialized to ones or Adam-style second-moment preconditioners. Clip or smooth dv_star to avoid numerical impulses. The paper supplies the hierarchical feedforward-plus-feedback architecture and bounded reference-trajectory principle; the gradient, Hessian, and disturbance statistics must be estimated empirically. The first cheap experiment should use a 2-layer MLP on MNIST or CIFAR-10, comparing SGD with momentum, Adam, and this optimizer at matched parameter-update budgets. Sweep h and K/M while logging loss, ||v-v_star||, maximum coordinate velocity, and gradient-noise sensitivity. The quantitative prediction is a sharp tracking-divergence boundary near h_c=2/lambda_max(M^{-1}K): measured error should decay for h<h_c and oscillate or diverge for h>h_c. In the stable regime, ||v_t-v_star_t|| should show geometric decay with slope approximately log rho(I-hM^{-1}K), while parameter updates should never exceed the imposed Vmax bound apart from discretization error.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.