Unverified 2026

Open-loop geodesic Frank–Wolfe for spherical weight blocks

Implementation & benchmark of arXiv:2608.21598 — Open-Loop Riemannian Frank--Wolfe: Fast Rates under Error Bounds and Scaling Inequalities

Usefulness6/10

Difficulty6/10

Novelty7/10

Optimization Stability Speedup Generalization Optimizer Training Initialization Geometry Optimization Differential geometry Convex analysis Dynamical systems

Source paper: Open-Loop Riemannian Frank--Wolfe: Fast Rates under Error Bounds and Scaling Inequalities arXiv:2608.21598 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace projected or retracted updates for constrained spherical parameter blocks with a geodesic Frank–Wolfe update and an iteration-only step size $\eta_k=a/(k+a)$. The method moves along a minimizing geodesic toward a feasible linear-oracle point, avoiding repeated projection and eliminating line-search or gap-feedback overhead. On locally error-bounded objectives, the paper predicts accelerated polynomial convergence, including $O(k^{-2})$ for strongly geodesically convex objectives and $O(k^{-1/(1-\theta)})$ under a Hölderian error bound.

Formulas

$$\eta_k=\frac{a}{k+a},\qquad a>2$$

$$s_k\in\operatorname*{arg\,min}_{s\in\mathcal C}\left\langle\operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s)\right\rangle_{x_k},\qquad x_{k+1}=\operatorname{Exp}_{x_k}\!\left(\eta_k\operatorname{Log}_{x_k}(s_k)\right)$$

$$G_k:=-\left\langle\operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s_k)\right\rangle_{x_k},\qquad f(x_k)-f^*=O\!\left(k^{-1/(1-\theta)}\right)$$

$$\alpha_R=\frac{1}{2}\cot R\quad (R<\pi/2),\qquad f(x_k)-f^*=O(k^{-2})\ \text{under strong geodesic convexity and an interior-ball condition}$$

Mathematical statement

The central schedule is $\eta_k=a/(k+a)$, where $k$ is the iteration index and $a>2$ is a fixed scalar. For a smooth geodesically convex objective $f$ on a feasible set $\mathcal C$ in a complete Riemannian manifold $\mathcal M$, choose a feasible linear-oracle point $s_k\in\mathcal C$ minimizing the local first-order model $\langle \operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s)\rangle_{x_k}$ over $s\in\mathcal C$, and update along a minimizing geodesic from $x_k$ to $s_k$. The geodesic interpolation is $x_{k+1}=\operatorname{Exp}_{x_k}(\eta_k\operatorname{Log}_{x_k}(s_k))$, where $\operatorname{Exp}_{x}$ is the Riemannian exponential map and $\operatorname{Log}_{x}=\operatorname{Exp}_{x}^{-1}$. The Frank–Wolfe gap is $G_k=-\langle \operatorname{grad}f(x_k),\operatorname{Log}_{x_k}(s_k)\rangle_{x_k}$. Under a local Hölderian error bound with exponent $\theta\in(0,1/2]$ and a local length-normalized directional scaling inequality, the paper states the eventual primal-error rate $f(x_k)-f^*=O(k^{-1/(1-\theta)})$ for every $a>2$, where $f^*=\min_{x\in\mathcal C}f(x)$. With an interior-ball condition and strong geodesic convexity, the rate improves to $O(k^{-2})$. For a geodesic ball of radius $R<\pi/2$ on the unit sphere, the scaling constant is explicitly $\alpha_R=\tfrac12\cot R$.

Implementation notes

Integrate this into training as an optimizer for parameter blocks constrained to a sphere or spherical geodesic ball, such as normalized classifier columns, hyperspherical embeddings, or a weight matrix whose columns must have fixed norm. Represent each block as $x\in\mathbb R^d$ with $\|x\|_2=1$, using the induced sphere geometry. At training step $k$, compute the Euclidean gradient $g=\nabla_x L$, project it to the tangent space as $g_T=g-\langle g,x\rangle x$, and choose a feasible oracle point $s_k$. For a full sphere, use the closed-form tangent descent endpoint $s_k=-g_T/\|g_T\|$ when it is numerically stable; for a spherical cap or geodesic ball, solve the small oracle by testing the cap boundary direction and interior candidate. Then compute $v=\operatorname{Log}_x(s_k)$, with sphere formula $\operatorname{Log}_x(s)=\frac{\phi}{\sin\phi}(s-\cos\phi\,x)$, $\phi=\arccos(\langle x,s\rangle)$, and update $x\leftarrow\operatorname{Exp}_x(\eta_k v)$, where $\operatorname{Exp}_x(v)=\cos(\|v\|)x+\sin(\|v\|)v/\|v\|$. Use $a=2.1,4,8$ as an ablation; do not tune $\eta_k$ from the current loss or gap. In code, clamp $\phi$ and use Taylor expansions near zero. Compute the paper's quantities exactly from the block gradient and geometry; estimate only the empirical rate by fitting a slope to $\log(f-f_{\min})$ and record the recent-window statistic $\min_{j\in\{\lceil k/2\rceil,\ldots,k\}}G_j$. First test on a small MLP or ViT classifier with a 128-dimensional hyperspherical classifier head on CIFAR-10, comparing AdamW plus normalization, projected SGD, and standard Riemannian gradient descent at equal forward/backward FLOPs. Measure training-loss descent, constraint violation, wall-clock time, gradient-norm stability, final accuracy, and calibration. The expected signal is lower optimizer overhead than projection or line search, stable exact norm preservation, and a log-log loss or gap slope approaching $-2$ on a synthetic strongly geodesically convex spherical quadratic; on real networks, treat any faster equal-FLOP descent or improved accuracy at the same norm constraint as evidence rather than assuming the theorem applies.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.