# Damped Spherical Momentum

- ID: 3087
- Canonical URL: https://synthcore.org/idea/3087/damped-spherical-momentum
- API JSON: https://synthcore.org/api/idea/3087.json
- API Markdown: https://synthcore.org/api/idea/3087.md
- Verification status: unverified
- Source: [arXiv:2609.02842](https://arxiv.org/abs/2609.02842)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: optimizer, training-dynamics, attention
- Math tags: dynamical-systems, geometry, optimization, control-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace unconstrained momentum updates for selected neural-network parameter blocks by damped second-order motion on a unit sphere. The update learns directions while preserving the block norm exactly, and includes the centripetal term required for a valid acceleration constrained to the sphere rather than merely projecting gradients after an unconstrained step.

## Mathematical statement

The paper defines the unit sphere $\mathcal{M}=\{\xi\in\mathcal{H}:\|\xi\|_{\mathcal{H}}=1\}$ and its tangent space $T_\xi\mathcal{M}=\{\eta\in\mathcal{H}:(\eta,\xi)_\mathcal{H}=0\}$. For a state $\theta(t)\in\mathcal{M}$, differentiating $\|\theta\|^2=1$ gives $(\theta,\dot\theta)=0$ and $(\theta,\ddot\theta)=-\|\dot\theta\|^2$. Let $P_\theta=I-\theta\theta^\top$ be the orthogonal projection onto $T_\theta\mathcal{M}$, let $g(\theta)=\nabla_\theta L(\theta)$ be the Euclidean loss gradient, let $v=\dot\theta$, and let $\gamma>0$ be damping. The constrained damped-gradient dynamics are $\ddot\theta+\gamma\dot\theta+g(\theta)=\lambda(t)\theta$, where the constraint gives $\lambda=\theta^\top g(\theta)+\|v\|^2$. Equivalently, $\ddot\theta=-P_\theta g(\theta)-\gamma v-\|v\|^2\theta$. If $L$ is restricted to the sphere and $E(t)=\frac12\|v\|^2+L(\theta)$, then $\frac{dE}{dt}=-\gamma\|v\|^2\le0$. The neural adaptation applies this construction independently to a parameter block normalized to radius $r$: use $u=\theta/r$, $P_u=I-uu^\top$, and rescale the force consistently by $r$.

## Key formulas

- $$\mathcal{M}=\{\xi\in\mathcal{H}:\|\xi\|_{\mathcal{H}}=1\},\qquad T_\xi\mathcal{M}=\{\eta\in\mathcal{H}:(\eta,\xi)_\mathcal{H}=0\}.$$
- $$\ddot\theta+\gamma\dot\theta+g(\theta)=\lambda\theta,\qquad \lambda=\theta^\top g(\theta)+\|\dot\theta\|^2,$$
- $$\ddot\theta=-\bigl(I-\theta\theta^\top\bigr)g(\theta)-\gamma\dot\theta-\|\dot\theta\|^2\theta,$$
- $$\frac{d}{dt}\left(\frac12\|\dot\theta\|^2+L(\theta)\right)=-\gamma\|\dot\theta\|^2.$$

## Implementation notes

(1) Integration point: apply this optimizer to one or more weight tensors, such as every Transformer attention projection matrix, each convolutional filter, or embedding row. Flatten a chosen block into $\theta\in\mathbb{R}^n$ and initialize or rescale it to a fixed radius $r$; keep a separate velocity tensor $v$ of the same shape. Do not constrain biases initially. The forward pass uses the original tensor $\theta$, so the architectural change is limited to optimizer state and update.

(2) Pseudocode:
```text
initialize theta with ||theta|| = r
v = 0
for each minibatch:
    loss = model(theta, batch)
    g = autograd(loss, theta)
    u = theta / r
    Pgrad = g - u * dot(u, g)
    a = -Pgrad - gamma * v - u * dot(v, v)
    v_half = v + h * a
    theta_trial = theta + h * v_half
    theta_new = r * theta_trial / ||theta_trial||
    u_new = theta_new / r
    v_new = v_half - u_new * dot(u_new, v_half)
    theta, v = theta_new, v_new
```
Here `Pgrad` is the tangent projection $P_u g$, the term `-u*dot(v,v)` is the centripetal correction from $\langle u,\ddot u\rangle=-\|v\|^2$, and the final projection enforces tangency after discretization. Use $h$ as the optimizer step size and $\gamma$ as damping; optionally use $v\leftarrow\beta v$ with $\beta=\exp(-\gamma h)$ for an exact damping substep.

(3) Computed versus estimated: the tangent projection, centripetal term, and norm/tangency checks are computed exactly from current tensors. No Hessian or PDE discretization is needed. Monitor $|\|\theta\|-r|$ and $|\theta^\top v|$; both should remain near floating-point precision after each step. The energy diagnostic is $E=\frac12\|v\|^2+L(\theta)$, although minibatch noise means it need not decrease every step. Tune $h$ and $\gamma$ on a logarithmic grid because excessive velocity can cause discrete-time oscillations.

(4) First cheap experiment: train a 4-layer MLP and a small ViT-Tiny on CIFAR-10, applying the method only to linear weight matrices. Compare SGD with momentum, AdamW, and AdamW followed by per-step weight normalization, using matched parameter counts and approximately matched optimizer-state memory. Run three seeds and record training loss versus optimizer steps and measured FLOPs, validation accuracy, gradient norm, norm drift, and cosine alignment $\theta^\top v$. The first success signal is equal-or-faster loss descent with substantially smaller norm drift and fewer exploding or oscillatory runs than momentum. A secondary signal is improved validation accuracy or robustness at matched training loss. Ablate the centripetal term and final tangent projection to verify that gains come from spherical dynamics rather than ordinary weight renormalization.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.