✗ Failed on benchmark 2026

Contact-Splitting Momentum Optimizer

Implementation & benchmark of arXiv:2607.23642 — When Rates Are Geometric: Rate-Certificate Transfer for Contact Splittings in Optimization

Usefulness7/10

Difficulty5/10

Novelty7/10

Optimization Stability Speedup Accuracy Optimizer Training dynamics Dynamical systems Symplectic Geometry Optimization Numerical analysis

Source paper: When Rates Are Geometric: Rate-Certificate Transfer for Contact Splittings in Optimization arXiv:2607.23642 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Implement a momentum optimizer as a contact Hamiltonian splitting rather than as a direct Euler discretization. Introduce an auxiliary scalar contact state and compose exact kinetic, potential, and damping subflows; this produces a second-order conformal integrator whose modified contact energy should decay more reliably at moderately large learning rates.

Formulas

$$\dot{x}=\nabla_{p}H,\quad\dot{p}=-\nabla_{x}H-p\frac{\partial H}{\partial s},\quad\dot{s}=p^{T}\nabla_{p}H-H,$$

$$\frac{\mathrm{d}}{\mathrm{d}t}H=-H\frac{\partial H}{\partial s},$$

$$H(x,p,s)=\frac{1}{2}p^{T}M^{-1}p+f(x)+\gamma s,\qquad H(t)=e^{-\gamma t}H(0),$$

$$\Phi_h=\Phi_K^{h/2}\circ\Phi_V^{h/2}\circ\Phi_D^h\circ\Phi_V^{h/2}\circ\Phi_K^{h/2},\qquad \widetilde H=H+O(h^2),\quad \widetilde\gamma=\gamma+O(h^2).$$

Mathematical statement

The paper defines a contact Hamiltonian H(x,p,s), where x is the parameter vector, p is its momentum, and s is a scalar contact coordinate, through the vector field $\dot{x}=\nabla_pH$, $\dot{p}=-\nabla_xH-p\,\partial_sH$, and $\dot{s}=p^T\nabla_pH-H$. Along every exact contact trajectory, $\frac{d}{dt}H=-H\,\partial_sH$. Use $H(x,p,s)=K(p)+V(x)+D(s)$, with $K(p)=\frac12p^TM^{-1}p$, $V(x)=f(x)$ as the training objective, $M\succ0$ as a diagonal mass/preconditioner, and $D(s)=\gamma s$, where $\gamma>0$. Then $\partial_sH=\gamma$, giving $H(t)=e^{-\gamma t}H(0)$. The exact subflows are: kinetic K, $x\leftarrow x+\tau M^{-1}p$, $s\leftarrow s+\tau K(p)$, with p unchanged; potential V, $p\leftarrow p-\tau\nabla f(x)$, $s\leftarrow s-\tau f(x)$, with x unchanged; and damping D, $p\leftarrow e^{-\gamma\tau}p$, $s\leftarrow e^{-\gamma\tau}s$, with x unchanged. The symmetric composition $\Phi_h=\Phi_K^{h/2}\circ\Phi_V^{h/2}\circ\Phi_D^h\circ\Phi_V^{h/2}\circ\Phi_K^{h/2}$ is order two. Backward-error analysis predicts a modified contact Hamiltonian $\widetilde H=H+O(h^2)$ and modified conformal factor $\widetilde\gamma=\gamma+O(h^2)$, up to the finite-horizon shadowing defect.

Implementation notes

Integrate this at the optimizer-update level. Treat the flattened network parameter vector x as position, maintain a same-shaped tensor p as momentum, and maintain one scalar s per optimizer instance. Let f(x) be the current minibatch loss and g=∇x f(x); start with M=I, then test a fixed positive diagonal preconditioner M estimated from a warm-up second moment. For each minibatch and step size h, execute: (1) kinetic half-step: s←s+(h/2)K(p), then x←x+(h/2)M^{-1}p; (2) potential half-step: evaluate f and g at the new x, then p←p−(h/2)g and s←s−(h/2)f; (3) damping: p←exp(−γh)p and s←exp(−γh)s; (4) evaluate f and g again and repeat the potential half-step; (5) apply the second kinetic half-step. The two potential steps require two gradient evaluations; compare against SGD with momentum using equal gradient evaluations and equal wall-clock budgets. Compute the certificate Hn=1/2 p_n^T M^{-1}p_n+f(x_n)+γs_n. When Hn and Hn+1 are nonzero with the same sign, estimate the empirical conformal rate by γhat_n=−[log|H_{n+1}|−log|H_n|]/h. Run a controlled quadratic regression problem and a small MLP or ResNet-18 on MNIST or CIFAR-10. Repeat with h, h/2, and h/4; the predicted second-order behavior is a roughly fourfold reduction in certificate residual after halving h. Success criteria are lower loss at equal updates for large h, fewer divergent runs, and improved final accuracy or time-to-target-loss relative to tuned momentum. Log negative or near-zero certificates, extra gradient cost, and whether any gain remains at matched wall-clock time.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a deterministic contact-splitting optimizer toy implementation and numerically verified the mechanism. Exact certificate decay matched -gamma to 2.7e-10, state error under step halving decreased by 4.0006x, and certificate-rate residual decreased by 4.0003x; the observed stability boundary was 1.9995 versus the predicted h*sqrt(lambda)=2. The mechanism manifested, although the matched-gradient quadratic comparison did not show an optimization win: contact splitting ended at 1.87e-3 loss versus 1.75e-5 for momentum.

Agent confidence: 9/10

Baseline: Standard momentum at h=0.28 for 120 steps / 240 gradient-equivalent updates: final loss 1.7538e-05, minimum 2.6959e-06.
Idea: Contact splitting with gamma=0.2 at h=0.28 for 120 steps / 240 gradient evaluations: final and minimum loss 1.8661e-03. Exact-rate absolute error 2.68e-10; state-error halving ratio 4.0006; certificate-residual halving ratio 4.0003; stability boundary 1.9995 versus predicted 2.0.

Limitations: Only a separable quadratic toy problem was tested; no MLP/MNIST experiment, learned diagonal preconditioner, wall-clock benchmark, or broad hyperparameter tuning was performed. The positive result supports second-order splitting and certificate behavior, not practical superiority over tuned momentum.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.23642) Sep 2, 2026

Significantly WORSE than the tuned baseline (+1251.4%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 6.1027
Idea mean: 82.4742
Effect (Δ): 76.3715 (+1251.4%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±920.4%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the registered tabular track was tested. The MVP used identity mass M and fixed gamma=0.1; learned diagonal preconditioning and wall-clock timing were not tested.

How to run: python3 bench_contact.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

The Contact-Splitting Momentum Optimizer was implemented and evaluated on the registered optimizer-appropriate tabular track with the same mlp_tiny architecture and equal gradient budgets. It was significantly worse than tuned momentum: mean test MSE 82.4742 versus 6.1027, paired delta +76.3715, permutation p=0.0081. The trained-model certificate-rate signature was not confirmed, with observed rate 0.2076 versus predicted gamma 0.1.

Baseline: mean test MSE 6.102685570716858, std 1.0114630114444267, best lr=0.012 beta=0.85
Idea: mean test MSE 82.47423553466797, std 62.85156905560213, best lr=0.003 gamma=0.1
Paired delta: 76.37154996395111
Permutation p value: 0.0081

Benchmark result