# Open-loop geodesic Frank–Wolfe for spherical weight blocks

- ID: 2674
- Canonical URL: https://synthcore.org/idea/2674/open-loop-geodesic-frank-wolfe-for-spherical-weight-blocks
- API JSON: https://synthcore.org/api/idea/2674.json
- API Markdown: https://synthcore.org/api/idea/2674.md
- Verification status: unverified
- Source: [arXiv:2608.21598](https://arxiv.org/abs/2608.21598)
- Category: optimization
- Solves: stability, speedup, generalization
- ML areas: optimizer, training, initialization
- Math tags: geometry, optimization, differential-geometry, convex-analysis, dynamical-systems
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Replace projected or retracted updates for constrained spherical parameter blocks with a geodesic Frank–Wolfe update and an iteration-only step size \(\eta_k=a/(k+a)\). The method moves along a minimizing geodesic toward a feasible linear-oracle point, avoiding repeated projection and eliminating line-search or gap-feedback overhead. On locally error-bounded objectives, the paper predicts accelerated polynomial convergence, including \(O(k^{-2})\) for strongly geodesically convex objectives and \(O(k^{-1/(1-\theta)})\) under a Hölderian error bound.

## Mathematical statement

The central schedule is \(\eta_k=a/(k+a)\), where \(k\) is the iteration index and \(a>2\) is a fixed scalar. For a smooth geodesically convex objective \(f\) on a feasible set \(\mathcal C\) in a complete Riemannian manifold \(\mathcal M\), choose a feasible linear-oracle point \(s_k\in\mathcal C\) minimizing the local first-order model \(\langle \operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s)\rangle_{x_k}\) over \(s\in\mathcal C\), and update along a minimizing geodesic from \(x_k\) to \(s_k\). The geodesic interpolation is \(x_{k+1}=\operatorname{Exp}_{x_k}(\eta_k\operatorname{Log}_{x_k}(s_k))\), where \(\operatorname{Exp}_{x}\) is the Riemannian exponential map and \(\operatorname{Log}_{x}=\operatorname{Exp}_{x}^{-1}\). The Frank–Wolfe gap is \(G_k=-\langle \operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s_k)\rangle_{x_k}\). Under a local Hölderian error bound with exponent \(\theta\in(0,1/2]\) and a local length-normalized directional scaling inequality, the paper states the eventual primal-error rate \(f(x_k)-f^*=O(k^{-1/(1-\theta)})\) for every \(a>2\), where \(f^*=\min_{x\in\mathcal C}f(x)\). With an interior-ball condition and strong geodesic convexity, the rate improves to \(O(k^{-2})\). For a geodesic ball of radius \(R<\pi/2\) on the unit sphere, the scaling constant is explicitly \(\alpha_R=\tfrac12\cot R\).

## Key formulas

- $$\eta_k=\frac{a}{k+a},\qquad a>2$$
- $$s_k\in\operatorname*{arg\,min}_{s\in\mathcal C}\left\langle\operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s)\right\rangle_{x_k},\qquad x_{k+1}=\operatorname{Exp}_{x_k}\!\left(\eta_k\operatorname{Log}_{x_k}(s_k)\right)$$
- $$G_k:=-\left\langle\operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s_k)\right\rangle_{x_k},\qquad f(x_k)-f^*=O\!\left(k^{-1/(1-\theta)}\right)$$
- $$\alpha_R=\frac{1}{2}\cot R\quad (R<\pi/2),\qquad f(x_k)-f^*=O(k^{-2})\ \text{under strong geodesic convexity and an interior-ball condition}$$

## Implementation notes

Integrate this into training as an optimizer for parameter blocks constrained to a sphere or spherical geodesic ball, such as normalized classifier columns, hyperspherical embeddings, or a weight matrix whose columns must have fixed norm. Represent each block as \(x\in\mathbb R^d\) with \(\|x\|_2=1\), using the induced sphere geometry. At training step \(k\), compute the Euclidean gradient \(g=\nabla_x L\), project it to the tangent space as \(g_T=g-\langle g,x\rangle x\), and choose a feasible oracle point \(s_k\). For a full sphere, use the closed-form tangent descent endpoint \(s_k=-g_T/\|g_T\|\) when it is numerically stable; for a spherical cap or geodesic ball, solve the small oracle by testing the cap boundary direction and interior candidate. Then compute \(v=\operatorname{Log}_x(s_k)\), with sphere formula \(\operatorname{Log}_x(s)=\frac{\phi}{\sin\phi}(s-\cos\phi\,x)\), \(\phi=\arccos(\langle x,s\rangle)\), and update \(x\leftarrow\operatorname{Exp}_x(\eta_k v)\), where \(\operatorname{Exp}_x(v)=\cos(\|v\|)x+\sin(\|v\|)v/\|v\|\). Use \(a=2.1,4,8\) as an ablation; do not tune \(\eta_k\) from the current loss or gap. In code, clamp \(\phi\) and use Taylor expansions near zero. Compute the paper's quantities exactly from the block gradient and geometry; estimate only the empirical rate by fitting a slope to \(\log(f-f_{\min})\) and record the recent-window statistic \(\min_{j\in\{\lceil k/2\rceil,\ldots,k\}}G_j\). First test on a small MLP or ViT classifier with a 128-dimensional hyperspherical classifier head on CIFAR-10, comparing AdamW plus normalization, projected SGD, and standard Riemannian gradient descent at equal forward/backward FLOPs. Measure training-loss descent, constraint violation, wall-clock time, gradient-norm stability, final accuracy, and calibration. The expected signal is lower optimizer overhead than projection or line search, stable exact norm preservation, and a log-log loss or gap slope approaching \(-2\) on a synthetic strongly geodesically convex spherical quadratic; on real networks, treat any faster equal-FLOP descent or improved accuracy at the same norm constraint as evidence rather than assuming the theorem applies.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.