Minimum-motion curvature-targeted preconditioner
Implementation & benchmark of arXiv:2607.07204 — Restricted Dynamic Geometric Complexity: Path-Space Reduction and Möbius--Jacobi Response
Source paper: Restricted Dynamic Geometric Complexity: Path-Space Reduction and Möbius--Jacobi Response arXiv:2607.07204 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace abrupt optimizer preconditioner changes with a metric trajectory that moves the smallest affine-invariant distance needed to reach a target generalized Hessian condition number. During training, optimize a short horizon of log-diagonal or block-SPD metrics using a terminal curvature penalty and an intrinsic kinetic regularizer, then execute only the first metric in a receding-horizon controller. The method should reduce oscillations caused by rapidly changing second-moment estimates while steering local optimization toward a better-conditioned regime.
Formulas
Mathematical statement
The paper defines the relative metric S=H^{-1/2}GH^{-1/2}, where H is a positive-definite Hessian or curvature matrix and G is a positive-definite optimizer metric. Its generalized condition target is kappa_gen(H,G)=lambda_max(G^{-1/2}HG^{-1/2})/lambda_min(G^{-1/2}HG^{-1/2})=kappa(S), and the acceptable endpoint set is C_K={S in S_{++}^d:kappa(S)<=K}, with K>=1. We use the affine-invariant SPD path length L(G_{0:T})=integral_0^T ||G(t)^{-1/2} dot{G}(t) G(t)^{-1/2}||_F dt and define D_K(G_0;H)=inf_{G(t):G(0)=G_0,G(T) in C_K} L(G_{0:T}). The paper states that fixed-horizon kinetic energy is exactly squared complexity divided by twice the horizon; therefore the discrete surrogate is A_hat=(1/(2T)) sum_{j=0}^{T-1} d_aff(G_j,G_{j+1})^2, where d_aff(A,B)=||log(A^{-1/2}BA^{-1/2})||_F. For diagonal metrics G_j=diag(exp(g_j)), this distance is exactly ||g_{j+1}-g_j||_2. The terminal condition uses M_T=G_T^{-1/2} H_hat G_T^{-1/2}, where H_hat is an estimated block Hessian, Gauss-Newton matrix, or Fisher matrix. A smooth violation penalty is [max(0,log(lambda_max(M_T)+epsilon)-log(lambda_min(M_T)+epsilon)-log K)]^2. The intrinsic penalty discourages coordinate-dependent multiplicative jumps, while the terminal penalty enforces the paper's spectral target.
Implementation notes
Integrate this mechanism at the optimizer-metric update, not in the network forward pass. Partition parameters into blocks of 256-2048 coordinates and initially use diagonal metrics, G_j=diag(exp(g_j)), because the affine-invariant distance then becomes an inexpensive squared Euclidean distance in log-metric coordinates. Maintain the current metric G_0 for each block and use the optimizer step theta <- theta - eta G_0^{-1} grad(theta). Every R training steps, estimate a block curvature H_hat using diagonal generalized Gauss-Newton or Fisher statistics; for small blocks, obtain lambda_max and lambda_min using 2-4 Hessian-vector products and a few Lanczos iterations. Set a planning horizon T=4 or 8 and initialize g_1,...,g_T from the current g_0. For 5-20 inner iterations, compute M_T=G_T^{-1/2}H_hatG_T^{-1/2}, estimate its extreme eigenvalues, form c=max(0,log(lambda_max+eps)-log(lambda_min+eps)-log K), and minimize beta*c^2+(alpha/(2T))*sum_j ||g_{j+1}-g_j||^2 by backpropagation through the planned metrics. Execute only g_1, then shift the plan and re-optimize at the next controller update. For full SPD blocks, parameterize G_j=G_{j-1}^{1/2}exp(V_j)G_{j-1}^{1/2}, with symmetric low-rank V_j, and use the matrix-log affine distance. The mathematical quantities are the generalized spectral target, affine-invariant distance, and kinetic coefficient; curvature estimates, K, alpha, beta, R, and T are empirical hyperparameters. First test on a 2-layer MLP or small ResNet on MNIST or CIFAR-10 against AdamW and a standard EMA diagonal preconditioner at equal optimizer-update budgets. Record loss versus steps, validation accuracy, gradient-norm variance, measured generalized condition number, metric jump size, and controller overhead. Success means faster loss descent or lower gradient variance at equal FLOPs without more than 10% wall-clock overhead; failure is no improvement in conditioning or convergence stability.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built a diagonal log-metric receding-horizon controller with terminal generalized-condition penalty and affine-invariant kinetic surrogate. The mathematical checks passed to numerical precision, and the controller reduced metric jumps by about 36% versus the Adam-style EMA control. However, across five paired noisy-quadratic runs it had worse mean conditioning, generally worse final loss, slightly higher gradient variance, and roughly 8x runtime, exceeding the claimed overhead budget; therefore the optimization win was not observed.
- Agent confidence
- 9/10
- Baseline
- Across seeds: final loss 0.0158-0.0320, mean generalized condition 1.38-1.67, metric jump 0.195-0.205, gradient variance 2.75-3.12, runtime 0.084-0.122 s.
- Idea
- Across seeds: final loss 0.0152-0.0417, mean generalized condition 4.64-5.51, metric jump 0.125-0.147, gradient variance 2.81-3.21, runtime 0.691-0.749 s. Math errors were approximately 2.2e-16 for diagonal affine distance, 2.2e-16 for affine invariance, and 6.9e-18 for the constant-speed kinetic identity.
Limitations: Only a small diagonal noisy quadratic was tested; no MLP, MNIST/CIFAR data, block-SPD metrics, Hessian-vector products, GPU execution, or wall-clock comparison on a neural network was evaluated. The curvature estimate was synthetic noisy diagonal curvature rather than an empirical neural-network Hessian or Fisher estimate.
How to run: python3 experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- experiment.py 4.9 KB View Raw
- report.md 1.6 KB View
- results.json 54.8 KB View Raw