# Bregman Newton momentum

- ID: 2662
- Canonical URL: https://synthcore.org/idea/2662/bregman-newton-momentum
- API JSON: https://synthcore.org/api/idea/2662.json
- API Markdown: https://synthcore.org/api/idea/2662.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.21359](https://arxiv.org/abs/2608.21359)
- Category: geometry
- Solves: accuracy, stability, generalization
- ML areas: moe-routing, optimizer, training-dynamics, attention
- Math tags: geometry, optimization, convex-analysis, information-theory
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Replace Euclidean momentum for selected neural parameters with a mirror or Bregman update, while using the paper's accelerated Newton direction for the objective step. Entropy geometry is especially suitable for softmax MoE routers, while Euclidean or log-barrier geometries can be used for unconstrained or positive parameters.

## Mathematical statement

For a differentiable strictly convex reference function d, the Bregman divergence is beta_d(u;v)=d(u)-d(v)-<nabla d(v),u-v>. The paper uses the identity beta_d(v_k;x^star)=beta_d(v_{k+1};x^star)+beta_d(v_k;v_{k+1})+<nabla d(v_k)-nabla d(v_{k+1}),v_{k+1}-x^star>, where the nonnegative term beta_d(v_k;v_{k+1}) supplies a telescoping stability potential. The mirror update is specified in dual coordinates by nabla d(v_{k+1})=nabla d(v_k)-a_{k+1}g_k, where g_k is a gradient or Newton-preconditioned gradient and a_{k+1} is the acceleration weight. For Euclidean d(v)=||v||^2/2 this is ordinary additive momentum. For entropy d(v)=sum_i(v_i log v_i-v_i), the inverse map gives v_{k+1,i}=v_{k,i} exp(-a_{k+1}g_{k,i}), followed by simplex normalization. The same cubic budget ell_k=a_{k+1}^3 L_2/A_{k+1}^2 can control the acceleration weight.

## Key formulas

- $$\beta_d(u;v)=d(u)-d(v)-\langle\nabla d(v),u-v\rangle$$
- $$\beta_d(v_k;x^\star)=\beta_d(v_{k+1};x^\star)+\beta_d(v_k;v_{k+1})+\langle\nabla d(v_k)-\nabla d(v_{k+1}),v_{k+1}-x^\star\rangle$$
- $$\nabla d(v_{k+1})=\nabla d(v_k)-a_{k+1}g_k$$
- $$d(v)=\sum_i v_i\log v_i-v_i\quad\Longrightarrow\quad v_{k+1,i}=v_{k,i}\exp(-a_{k+1}g_{k,i})$$

## Implementation notes

(1) Integration point: modify the optimizer state for parameters with natural non-Euclidean geometry, initially the softmax router logits or probabilities in a small mixture-of-experts classifier. Compute the accelerated extrapolation y_k, the minibatch gradient q_k, and optionally a Newton-preconditioned direction g_k by solving (H_k+lambda I)g_k=q_k with CG. (2) Pseudocode: maintain dual state z_k=nabla d(v_k). Form y_k=(A_k x_k+a_{k+1}v_k)/A_{k+1}; compute q_k and g_k; update z_new=z_k-a_{k+1}g_k; recover v_new=nabla d^*(z_new); set x_new=y_k-g_k and continue. For an MoE router use entropy geometry: compute temporary probabilities p_new proportional to p_old*exp(-a*g), normalize across experts, and convert back to logits if the implementation stores logits. Clip or reject a step when beta_d(v_k;v_new) exceeds a fixed fraction c of the current loss. (3) Compute beta_d exactly from d. For entropy, use epsilon-clipped probabilities and log-sum-exp normalization. Estimate L_2 with random Hessian-vector probes and choose a_{k+1} by ell_k R_hat_k <= eta, where R_hat_k is an EMA of the Bregman displacement or parameter displacement. (4) First experiment: train a small MoE classifier on CIFAR-10 with fixed experts initially, comparing AdamW, Euclidean accelerated Newton, and entropy-Bregman accelerated Newton for the router. Then unfreeze all parameters. Record router entropy, expert load variance, dropped-token rate, loss per Hessian-vector product, and accuracy. Success means fewer routing collapses and faster convergence at equal optimizer compute.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built and ran a NumPy MVP for entropy mirror updates, Bregman identity verification, local stability sweeps, and a synthetic fixed-expert router comparison. The stated three-point identity failed badly, with maximum absolute residual 0.2763, while valid small-step KL scaling matched prediction closely with ratios 1.00015–1.00664. Mirror descent preserved higher entropy and lower load variance but converged slower than Adam, so the proposed mechanism did not demonstrate the claimed overall win.

### Mechanism check

- Verdict: Built and ran a NumPy MVP for entropy mirror updates, Bregman identity verification, local stability sweeps, and a synthetic fixed-expert router comparison. The stated three-point identity failed badly, with maximum absolute residual 0.2763, while valid small-step KL scaling matched prediction closely with ratios 1.00015–1.00664. Mirror descent preserved higher entropy and lower load variance but converged slower than Adam, so the proposed mechanism did not demonstrate the claimed overall win.
- Confidence: 9/10
- Limitations: The experiment was synthetic rather than CIFAR-10 or a trainable neural MoE. It did not implement accelerated Newton directions, cubic-budget step selection, clipping or rejection, Hessian-vector products, or unfreezing experts. The supplied Bregman identity was tested as written and was not replaced with an alternative convention.

## Artifacts

- [experiment.py](https://synthcore.org/code/1019/experiment.py)
- [report.md](https://synthcore.org/code/1019/report.md)
- [results.json](https://synthcore.org/code/1019/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1019)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.