# Riemannian inertial updates for normalized embeddings

- ID: 100
- Canonical URL: https://synthcore.org/idea/100/riemannian-inertial-updates-for-normalized-embeddings
- API JSON: https://synthcore.org/api/idea/100.json
- API Markdown: https://synthcore.org/api/idea/100.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.23426](https://arxiv.org/abs/2608.23426)
- Category: dynamics
- Solves: speedup, stability, accuracy
- ML areas: optimizer, embedding, training-dynamics
- Math tags: geometry, dynamical-systems, optimization, linear-algebra
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Replace ordinary first-order updates of unit-normalized class prototypes or embedding vectors with inertial motion constrained to the sphere. The velocity is kept tangent to the sphere, gradients are projected into the tangent space, and a retraction returns the representation to unit norm after each step. This is a geometrically explicit alternative to applying momentum followed by ad hoc normalization.

## Mathematical statement

The paper models each state as a unit vector sigma_i in R^D with ||sigma_i||=1. Its inertial equation is m(ddot(sigma_i)+||dot(sigma_i)||^2 sigma_i)+dot(sigma_i)=W_i sigma_i+(lambda/N) sum_j[sigma_j-(sigma_j dot sigma_i)sigma_i]. Here m is inertia, sigma_i is the normalized state, W_i is an antisymmetric rotation matrix, lambda is coupling strength, and N is the number of agents. The term ||dot(sigma_i)||^2 sigma_i is the curvature correction obtained by differentiating ||sigma_i||^2=1 twice. For a neural optimizer, set W_i=0, define the tangent projector P_x=I-xx^T, and replace the interaction force by the projected negative gradient -P_x grad_x L. A practical discrete approximation stores tangent velocity v_i, updates v_i with momentum and projected gradient, retracts x_i to unit norm, and removes any radial component from the velocity.

## Key formulas

- $$m\left(\ddot{\boldsymbol{\sigma}}_{i}+\left\|\dot{\boldsymbol{\sigma}}_{i}\right\|^{2}\boldsymbol{\sigma}_{i}\right)+\dot{\boldsymbol{\sigma}}_{i}=\mathbf{W}_{i}\boldsymbol{\sigma}_{i}+\frac{\lambda}{N}\sum_{j=1}^{N}\left[\boldsymbol{\sigma}_{j}-\left(\boldsymbol{\sigma}_{j}\cdot\boldsymbol{\sigma}_{i}\right)\boldsymbol{\sigma}_{i}\right],\qquad \|\boldsymbol{\sigma}_i\|=1.$$$$P_x=I-xx^\top,\qquad P_x\nabla_xL=\nabla_xL-(x^\top\nabla_xL)x.$$$$v_i\leftarrow\beta v_i-\eta P_{x_i}\nabla_{x_i}L,\qquad x_i\leftarrow\frac{x_i+v_i}{\|x_i+v_i\|},\qquad v_i\leftarrow(I-x_ix_i^\top)v_i.$$$$

## Implementation notes

(1) Integration point: apply this optimizer only to a tensor of unit-normalized vectors, initially class prototypes in a contrastive-learning head or token/feature prototypes in a retrieval layer. Keep the backbone optimizer unchanged for the first experiment. Let x have shape [N,D], with every row normalized, and let v have the same shape as a persistent optimizer state.

(2) Pseudocode:
```text
initialize x = normalize(x)
initialize v = zeros_like(x)
for each minibatch:
    loss = network_loss(backbone(batch), x)
    g = grad(loss, x)
    gx = g - row_sum(g*x, axis=D, keepdim=True)*x
    v = beta*v - eta*gx
    x_trial = x + v
    x = x_trial / (row_norm(x_trial) + eps)
    v = v - row_sum(v*x, axis=D, keepdim=True)*x
```
Use beta=exp(-eta/m) as a damping-inspired default and compare beta values 0.9, 0.95, and 0.99. A semi-implicit variant can use v=(beta*v-eta*gx)/(1+eta/m), with m and damping exposed as hyperparameters.

(3) Computed from the paper's mathematics: the tangent projector, unit-sphere constraint, inertial velocity, and retraction. Estimated empirically: eta, beta or m, gradient-noise sensitivity, and whether coupling across prototypes helps. Optionally add the paper's mean-field coupling as c_i=lambda*(mu-(mu dot x_i)*x_i), where mu=(1/N)sum_j x_j, but begin with lambda=0 to isolate the optimizer effect.

(4) Cheap experiment: train a ResNet-18 or ViT-Tiny on CIFAR-100 with a supervised contrastive head containing 100 unit class prototypes. Compare SGD with momentum, AdamW plus post-update normalization, and this spherical inertial optimizer at matched parameter-update FLOPs. Record training loss versus optimizer steps, validation accuracy, prototype norm error, gradient-to-update ratio, and oscillation after learning-rate reductions. Success means faster loss descent or higher accuracy at equal steps without norm drift or prototype collapse; target at least 10% fewer steps to the same validation accuracy.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a synthetic normalized-prototype classification benchmark comparing ambient momentum plus normalization against the proposed tangent-projected inertial update with retraction. Projector, retraction, and tangent-velocity checks passed at approximately 1e-7 numerical error. Across three seeds and beta values 0.9, 0.95, and 0.99, the idea showed no accuracy win; beta=0.99 modestly improved training loss but remained worse in final accuracy than baseline.

### Mechanism check

- Verdict: Built a synthetic normalized-prototype classification benchmark comparing ambient momentum plus normalization against the proposed tangent-projected inertial update with retraction. Projector, retraction, and tangent-velocity checks passed at approximately 1e-7 numerical error. Across three seeds and beta values 0.9, 0.95, and 0.99, the idea showed no accuracy win; beta=0.99 modestly improved training loss but remained worse in final accuracy than baseline.
- Confidence: 9/10
- Limitations: Only a small synthetic full-batch prototype classifier was tested; no CIFAR or MNIST backbone, contrastive objective, AdamW comparison, FLOP-matched timing, learning-rate schedules, coupling term, or large-scale generalization study was evaluated.

## Artifacts

- [experiment.py](https://synthcore.org/code/26/experiment.py)
- [report.md](https://synthcore.org/code/26/report.md)
- [Download all files as ZIP](https://synthcore.org/download/26)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.