△ Mechanism confirmed, baseline not beaten 2026

Affine-Invariant Kronecker Preconditioner

Implementation & benchmark of arXiv:2607.12286 — Structured Preconditioning in Affine-Invariant Geometry: Projection, Certificates, and Kronecker Separation

Usefulness8/10

Difficulty6/10

Novelty6/10

Optimization Speedup Stability Memory Optimizer Mlp Transformer Geometry Optimization Linear algebra

Source paper: Structured Preconditioning in Affine-Invariant Geometry: Projection, Certificates, and Kronecker Separation arXiv:2607.12286 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace Euclidean or entrywise Kronecker fitting of a layer curvature matrix with its affine-invariant projection onto G = A tensor B. Use the resulting factors as a compact SPD preconditioner in the optimizer, while solving the projection through logarithmic residual partial traces and Armijo line search.

Formulas

$$d_{\mathrm{AI}}(G_{0},G_{1})=\left\lVert\log(G_{0}^{-1/2}G_{1}G_{0}^{-1/2})\right\rVert_{F}.$$

$$G^*=\arg\min_{A\succ0,\,B\succ0}\left\|\log\!\left((A\otimes B)^{-1/2}H(A\otimes B)^{-1/2}\right)\right\|_F^2.$$

$$R=\log\!\left(G^{-1/2}HG^{-1/2}\right),\qquad \operatorname{Tr}_{q}(R)=0,\quad \operatorname{Tr}_{p}(R)=0.$$

$$(A\otimes B)^{-1}\operatorname{vec}(X)=\operatorname{vec}(B^{-1}XA^{-T}).$$

Mathematical statement

For an SPD target H in S++^(pq) and candidate SPD geometry G in S++^(pq), the paper measures mismatch by d_AI(G0,G1) = ||log(G0^(-1/2) G1 G0^(-1/2))||_F. The structured family is K = {A tensor B : A in S++^p, B in S++^q}. Its affine-invariant projection is G* = argmin over G in K of d_AI(G,H)^2, and geodesic convexity gives uniqueness. Define the normalized logarithmic residual R = log(G^(-1/2) H G^(-1/2)). In tensor coordinates, projection stationarity is expressed by vanishing partial traces, Tr_q(R) = 0 and Tr_p(R) = 0, modulo the scalar gauge A tensor B = (cA) tensor (B/c). Here Tr_q sums the q by q diagonal blocks and returns a p by p matrix, while Tr_p returns a q by q matrix. The structured inverse can be applied without materializing G: (A tensor B)^(-1) vec(X) = vec(B^(-1) X A^(-T)).

Implementation notes

Integrate this into Adam or a K-FAC-like optimizer for a linear or attention projection layer W in R^(p by q). Every K optimizer steps, estimate an SPD curvature target H from a damped gradient covariance, a block Hessian-vector approximation, or a Fisher matrix: H = (1/m) sum_i g_i g_i^T + lambda I, with lambda greater than zero. Maintain SPD factors A and B, initialize them from the previous iteration or identity, and impose the gauge log det A = 0 after every update. Form G = A tensor B, compute R = log(G^(-1/2) H G^(-1/2)) using an eigendecomposition for small MVPs or a matrix-log routine, then compute both partial traces. Update the factors in SPD-preserving coordinates, for example A <- A^(1/2) exp(alpha D_A) A^(1/2) and B <- B^(1/2) exp(alpha D_B) B^(1/2), where D_A and D_B are whitened versions of the partial-trace residuals. Choose alpha by Armijo backtracking on ||R||_F^2 and stop when both partial-trace norms are small. The exact paper-derived quantities are the affine-invariant residual and partial-trace stationarity test; damping, stochastic curvature estimation, and the factor update are engineering choices. During training, replace the raw matrix gradient by G^(-1)g, implemented as B^(-1) X A^(-T) after reshaping the vector gradient into X. First test on a two-layer MLP on CIFAR-10 with p and q in {64, 128}, comparing SGD, AdamW, Euclidean Kronecker fitting, and affine-invariant projection at equal wall-clock and memory budgets. Measure training loss versus optimizer steps, loss versus FLOPs, final accuracy, factor-update overhead, and the condition number of the preconditioned empirical Hessian. Success means faster loss descent or a smaller preconditioned condition number at comparable memory; failure is no improvement over Euclidean fitting after accounting for projection cost.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built an affine-invariant Kronecker projection MVP with SPD log/exp factors, logarithmic residual partial traces, Euclidean Kronecker fitting, structured inverse application, and quadratic preconditioning tests. The mechanism manifested: transverse perturbation distance tracked epsilon with observed/predicted ratios from 0.990 to 0.991, partial-trace residuals stayed below 3.2e-8, scale-invariant targets had residuals below 3.2e-8, and the inverse identity error was 4.4e-15. Across 8 random 2x2 trials, affine projection reduced mean relative condition number from 44.31 for Euclidean fitting to 19.02 and had lower step-20 loss in 7/8 trials, although projection required roughly 10-14 numerical iterations.

Agent confidence: 8/10

Baseline: Euclidean Kronecker fitting: mean relative condition number 44.31 and mean step-20 quadratic loss 0.04723.
Idea: Affine-invariant projection: mean relative condition number 19.02 and mean step-20 quadratic loss 0.004260; lower condition number than Euclidean fitting in 8/8 trials and lower step-20 loss in 7/8 trials. Math checks: distances 0.00990, 0.02970, 0.09901, 0.29704, and 0.99046 for epsilon 0.01, 0.03, 0.1, 0.3, and 1.0 versus the predicted epsilon; partial-trace norms <=3.2e-8; inverse relative error 4.43e-15.

Limitations: This is a small NumPy/SciPy quadratic experiment only with p=q=2, dense eigendecompositions, and BFGS factor fitting. It does not test CIFAR-10, MLP training, Adam/K-FAC integration, wall-clock or FLOP parity, stochastic curvature estimation, larger dimensions, or a full Armijo factor-update implementation.

How to run: python3 affine_kron_mvp.py && /home/maxwelhelp/main/bin/python3 extended_check.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.12286) Sep 2, 2026

Significantly WORSE than the tuned baseline (+44.6%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 4.8651
Idea mean: 7.0341
Effect (Δ): 2.1689 (+44.6%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±13.5%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 4 configs

Limitations:

Only tabular/mlp_tiny was tested for 15 epochs. The MVP uses diagonal rather than dense A and B factors and does not implement full dense Armijo projection; wall-clock/FLOPs and larger dimensions were not evaluated.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 10/10

On the structurally matched tabular track with shared mlp_tiny architecture, tuned Adam achieved mean test MSE 4.865146 while the best affine-invariant diagonal Kronecker setting achieved 7.034068. The paired delta was +2.168922 with permutation p=0.0081, so the idea was significantly worse despite improving the measured curvature-conditioning signature.

Baseline: Adam; best cfg lr=0.01, weight_decay=0.0; mean test MSE 4.865146; std 0.489887; 8 seeds.
Idea: Affine-invariant diagonal Kronecker preconditioner; best cfg lr=0.01, weight_decay=0.0; mean test MSE 7.034068; 0/8 paired wins; delta +2.168922; p=0.0081.

Benchmark result