# Minimum-motion curvature-targeted preconditioner

- ID: 514
- Canonical URL: https://synthcore.org/idea/514/minimum-motion-curvature-targeted-preconditioner
- API JSON: https://synthcore.org/api/idea/514.json
- API Markdown: https://synthcore.org/api/idea/514.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.07204](https://arxiv.org/abs/2607.07204)
- Category: optimization
- Solves: stability, speedup, accuracy
- ML areas: optimizer, training-dynamics
- Math tags: geometry, optimization, linear-algebra, dynamical-systems
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace abrupt optimizer preconditioner changes with a metric trajectory that moves the smallest affine-invariant distance needed to reach a target generalized Hessian condition number. During training, optimize a short horizon of log-diagonal or block-SPD metrics using a terminal curvature penalty and an intrinsic kinetic regularizer, then execute only the first metric in a receding-horizon controller. The method should reduce oscillations caused by rapidly changing second-moment estimates while steering local optimization toward a better-conditioned regime.

## Mathematical statement

The paper defines the relative metric S=H^{-1/2}GH^{-1/2}, where H is a positive-definite Hessian or curvature matrix and G is a positive-definite optimizer metric. Its generalized condition target is kappa_gen(H,G)=lambda_max(G^{-1/2}HG^{-1/2})/lambda_min(G^{-1/2}HG^{-1/2})=kappa(S), and the acceptable endpoint set is C_K={S in S_{++}^d:kappa(S)<=K}, with K>=1. We use the affine-invariant SPD path length L(G_{0:T})=integral_0^T ||G(t)^{-1/2} dot{G}(t) G(t)^{-1/2}||_F dt and define D_K(G_0;H)=inf_{G(t):G(0)=G_0,G(T) in C_K} L(G_{0:T}). The paper states that fixed-horizon kinetic energy is exactly squared complexity divided by twice the horizon; therefore the discrete surrogate is A_hat=(1/(2T)) sum_{j=0}^{T-1} d_aff(G_j,G_{j+1})^2, where d_aff(A,B)=||log(A^{-1/2}BA^{-1/2})||_F. For diagonal metrics G_j=diag(exp(g_j)), this distance is exactly ||g_{j+1}-g_j||_2. The terminal condition uses M_T=G_T^{-1/2} H_hat G_T^{-1/2}, where H_hat is an estimated block Hessian, Gauss-Newton matrix, or Fisher matrix. A smooth violation penalty is [max(0,log(lambda_max(M_T)+epsilon)-log(lambda_min(M_T)+epsilon)-log K)]^2. The intrinsic penalty discourages coordinate-dependent multiplicative jumps, while the terminal penalty enforces the paper's spectral target.

## Key formulas

- $$S=H^{-1/2}GH^{-1/2},\qquad \kappa_{\rm gen}(H,G)=\frac{\lambda_{\max}(G^{-1/2}HG^{-1/2})}{\lambda_{\min}(G^{-1/2}HG^{-1/2})}=\kappa(S).$$
- $$\mathcal{C}_{K}=\{S\in\mathbb{S}_{++}^{d}:\kappa(S)\leq K\},\qquad D_{K,\mathfrak F}(S_0;H)=\inf_{S(0)=S_0,\ S(T)\in\mathcal C_K}\int_0^T\left\|S(t)^{-1/2}\dot S(t)S(t)^{-1/2}\right\|_Fdt.$$
- $$A_T^\star=\frac{D_{K,\mathfrak F}(S_0;H)^2}{2T},\qquad \widehat A_T=\frac{1}{2T}\sum_{j=0}^{T-1}d_{\rm aff}(G_j,G_{j+1})^2,\quad d_{\rm aff}(A,B)=\left\|\log\left(A^{-1/2}BA^{-1/2}\right)\right\|_F.$$
- $$\min_{G_1,\ldots,G_T}\ \beta\,\big[\max(0,\log\widehat\kappa_T-\log K)\big]^2+\frac{\alpha}{2T}\sum_{j=0}^{T-1}d_{\rm aff}(G_j,G_{j+1})^2,\quad \widehat\kappa_T=\frac{\lambda_{\max}(G_T^{-1/2}\widehat H_TG_T^{-1/2})}{\lambda_{\min}(G_T^{-1/2}\widehat H_TG_T^{-1/2})}.$$

## Implementation notes

Integrate this mechanism at the optimizer-metric update, not in the network forward pass. Partition parameters into blocks of 256-2048 coordinates and initially use diagonal metrics, G_j=diag(exp(g_j)), because the affine-invariant distance then becomes an inexpensive squared Euclidean distance in log-metric coordinates. Maintain the current metric G_0 for each block and use the optimizer step theta <- theta - eta G_0^{-1} grad(theta). Every R training steps, estimate a block curvature H_hat using diagonal generalized Gauss-Newton or Fisher statistics; for small blocks, obtain lambda_max and lambda_min using 2-4 Hessian-vector products and a few Lanczos iterations. Set a planning horizon T=4 or 8 and initialize g_1,...,g_T from the current g_0. For 5-20 inner iterations, compute M_T=G_T^{-1/2}H_hatG_T^{-1/2}, estimate its extreme eigenvalues, form c=max(0,log(lambda_max+eps)-log(lambda_min+eps)-log K), and minimize beta*c^2+(alpha/(2T))*sum_j ||g_{j+1}-g_j||^2 by backpropagation through the planned metrics. Execute only g_1, then shift the plan and re-optimize at the next controller update. For full SPD blocks, parameterize G_j=G_{j-1}^{1/2}exp(V_j)G_{j-1}^{1/2}, with symmetric low-rank V_j, and use the matrix-log affine distance. The mathematical quantities are the generalized spectral target, affine-invariant distance, and kinetic coefficient; curvature estimates, K, alpha, beta, R, and T are empirical hyperparameters. First test on a 2-layer MLP or small ResNet on MNIST or CIFAR-10 against AdamW and a standard EMA diagonal preconditioner at equal optimizer-update budgets. Record loss versus steps, validation accuracy, gradient-norm variance, measured generalized condition number, metric jump size, and controller overhead. Success means faster loss descent or lower gradient variance at equal FLOPs without more than 10% wall-clock overhead; failure is no improvement in conditioning or convergence stability.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a diagonal log-metric receding-horizon controller with terminal generalized-condition penalty and affine-invariant kinetic surrogate. The mathematical checks passed to numerical precision, and the controller reduced metric jumps by about 36% versus the Adam-style EMA control. However, across five paired noisy-quadratic runs it had worse mean conditioning, generally worse final loss, slightly higher gradient variance, and roughly 8x runtime, exceeding the claimed overhead budget; therefore the optimization win was not observed.

### Mechanism check

- Verdict: Built a diagonal log-metric receding-horizon controller with terminal generalized-condition penalty and affine-invariant kinetic surrogate. The mathematical checks passed to numerical precision, and the controller reduced metric jumps by about 36% versus the Adam-style EMA control. However, across five paired noisy-quadratic runs it had worse mean conditioning, generally worse final loss, slightly higher gradient variance, and roughly 8x runtime, exceeding the claimed overhead budget; therefore the optimization win was not observed.
- Confidence: 9/10
- Limitations: Only a small diagonal noisy quadratic was tested; no MLP, MNIST/CIFAR data, block-SPD metrics, Hessian-vector products, GPU execution, or wall-clock comparison on a neural network was evaluated. The curvature estimate was synthetic noisy diagonal curvature rather than an empirical neural-network Hessian or Fisher estimate.

## Artifacts

- [experiment.py](https://synthcore.org/code/1365/experiment.py)
- [report.md](https://synthcore.org/code/1365/report.md)
- [results.json](https://synthcore.org/code/1365/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1365)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.