# Curvature-scaled coordinate trust-region momentum

- ID: 67
- Canonical URL: https://synthcore.org/idea/67/curvature-scaled-coordinate-trust-region-momentum
- API JSON: https://synthcore.org/api/idea/67.json
- API Markdown: https://synthcore.org/api/idea/67.md
- Verification status: unverified
- Source: [arXiv:2608.22575](https://arxiv.org/abs/2608.22575)
- Category: optimization
- Solves: stability, speedup, accuracy
- ML areas: graph-nn, optimizer, training-dynamics
- Math tags: optimization, numerical-analysis, linear-algebra
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Replace an unconstrained adaptive optimizer step with a coordinate-wise trust-region step whose radius is determined by the current gradient and accumulated AdaGrad weight, then damp it using a curvature estimate and momentum. This can prevent unstable updates on sparse or highly anisotropic GNN parameters while retaining aggressive movement on coordinates with reliable gradients.

## Mathematical statement

At iteration \(k\), \(\bm g_k\) is the current or subsampled gradient, \(\bm w_k\) is the accumulated coordinate-wise AdaGrad weight, and the trust-region radius for coordinate \(i\) is \(\Delta_{k,i}=|g_{k,i}|/w_{k,i}\). Let \(\bm s_k^S\) be a proposed search step and \(\bm B_k\) a symmetric curvature or Hessian approximation. The paper's scalar correction is \(\alpha_k=\min(1,-\langle\bm g_k,\bm s_k^S\rangle/\langle\bm s_k^S,\bm B_k\bm s_k^S\rangle)\) when the denominator is positive, and \(\alpha_k=1\) otherwise. A momentum state uses \(\bm m_k=\beta\bm m_{k-1}+(1-\beta)\bm s_k^Q\), after which every coordinate is clipped to \([-\Delta_{k,i},\Delta_{k,i}]\). Here \(\bm s_k^Q=\alpha_k\bm s_k^S\), \(\beta\in[0,1)\), and \(\bm s_k^S\) can be a preconditioned negative-gradient step. The denominator measures predicted quadratic curvature; clipping enforces a coordinate trust region.

## Key formulas

- $$\bm w_{k,i}=\sqrt{\sum_{t=1}^{k}g_{t,i}^{2}}+\varepsilon,\qquad \Delta_{k,i}=\frac{|g_{k,i}|}{w_{k,i}}.$$
- $$\alpha_k=\begin{cases}\min\left(1,\dfrac{-\langle\bm g_k,\bm s_k^S\rangle}{\langle\bm s_k^S,\bm B_k\bm s_k^S\rangle}\right),&\text{if }\langle\bm s_k^S,\bm B_k\bm s_k^S\rangle>0,\\1,&\text{otherwise},\end{cases}$$
- $$\bm m_k=\beta\bm m_{k-1}+(1-\beta)\bm s_k^Q,\qquad \bm s_k^Q=\alpha_k\bm s_k^S,$$
- $$m_{k,i}\leftarrow\min\bigl(\max(m_{k,i},-\Delta_{k,i}),\Delta_{k,i}\bigr),\qquad \bm\theta_k=\bm\theta_{k-1}+\bm m_k.$$

## Implementation notes

(1) Integration point: implement this as a drop-in replacement for the parameter-update portion of SGD or AdaGrad in a GNN, using the same gradient tensors and parameter groups. It can be applied after every local or coarse gradient computation from the two-level method, or tested independently on a full-graph baseline.

(2) Pseudocode:
```text
input theta, m = 0, w = 0
for k = 1,...,K:
    g = gradient(loss(theta))
    w = sqrt(w*w + g*g) + eps
    sS = -eta * g / w
    B_s = diagonal_curvature(sS)
    q = dot(sS, B_s)
    if q > 0: alpha = min(1, -dot(g,sS)/q)
    else:     alpha = 1
    sQ = alpha * sS
    m = beta*m + (1-beta)*sQ
    m = clamp(m, -abs(g)/w, abs(g)/w)
    theta = theta + m
```
(3) The accumulator, trust radius, curvature ratio, momentum, and clipping are computed from the displayed equations. For an inexpensive curvature model, use \(B_{k,i}=g_{k,i}^2+\lambda\) as a diagonal Fisher proxy, or periodically estimate \(\langle s_k^S,B_ks_k^S\rangle\) with one Hessian-vector product. The mathematical ratio is exact for the chosen proxy; its usefulness must be measured empirically. (4) First cheap experiment: compare this optimizer against AdamW, AdaGrad, and the unmodified AG2m-style update on a 2-layer GCN and GraphSAGE using ogbn-arxiv or Cora. Keep batch size and gradient evaluations fixed. Plot training loss, validation accuracy, gradient norm, curvature ratio, fraction of clipped coordinates, and wall-clock time. Success means fewer divergence spikes, faster loss decrease at equal FLOPs, and equal-or-better validation accuracy. Ablate curvature correction, momentum, and clipping separately.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.