✗ Failed on benchmark 2026

Extended-Symplectic Neural Optimizer

Implementation & benchmark of arXiv:2607.12914 — Discrete-time generalized canonical transformations for non-autonomous systems

Usefulness8/10

Difficulty5/10

Novelty6/10

Dynamics Stability Accuracy Generalization Optimizer Training dynamics Scheduler Dynamical systems Control theory Numerical analysis Optimization Linear algebra

Source paper: Discrete-time generalized canonical transformations for non-autonomous systems arXiv:2607.12914 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a dissipative optimizer update with a canonical discrete flow on the extended state $(\theta,p,t,e)$, where $\theta$ are network parameters, $p$ is momentum, $t$ is training time, and $e$ is its conjugate energy variable. Use a symmetric composition of exact Hamiltonian subflows for kinetic energy, loss, and time translation; this preserves the extended symplectic form and avoids artificial phase-volume collapse. Weak restarts or occasional damping can be added separately if convergence rather than conservative exploration is required.

Formulas

$$H_{\mathrm{ext}}(\theta,p,t,e)=e+\frac12p^{\mathsf T}M^{-1}p+L(\theta,t),\qquad \Omega=d\theta\wedge dp+dt\wedge de.$$

$$A_{\tau}:\ t\leftarrow t+\tau;\qquad B_{\tau}:\ \theta\leftarrow\theta+\tau M^{-1}p;\qquad C_{\tau}:\ p\leftarrow p-\tau\nabla_{\theta}L(\theta,t),\quad e\leftarrow e-\tau\partial_tL(\theta,t).$$

$$\Psi_h=A_{h/2}\circ B_{h/2}\circ C_h\circ B_{h/2}\circ A_{h/2},\qquad (D\Psi_h)^{\mathsf T}J(D\Psi_h)=J,\quad \det(D\Psi_h)=1,$$ $$J=\begin{bmatrix}0&I&0&0\\-I&0&0&0\\0&0&0&1\\0&0&-1&0\end{bmatrix}.$$

$$h<h_{\mathrm{crit}}=\frac{2}{\sqrt{\lambda_{\max}(M^{-1}K)}}\quad\Longleftrightarrow\quad h^2\lambda_{\max}(M^{-1}K)<4.$$

Mathematical statement

Let $\theta\in\mathbb{R}^{d}$ be neural-network parameters, $p\in\mathbb{R}^{d}$ canonical momentum, $t\in\mathbb{R}$ training time, $e\in\mathbb{R}$ conjugate time momentum, $M\succ0$ a mass matrix, and $L(\theta,t)$ the loss. Define $H_{\mathrm{ext}}(\theta,p,t,e)=e+T(p)+L(\theta,t)$ with $T(p)=\frac12p^{\mathsf T}M^{-1}p$. The extended canonical form is $\Omega=d\theta\wedge dp+dt\wedge de$. Exact subflows of the separable Hamiltonian terms are symplectomorphisms, so their composition satisfies $(D\Psi_h)^{\mathsf T}J(D\Psi_h)=J$ and has determinant one. For a locally quadratic loss with Hessian $K\succeq0$, the symmetric position-momentum update is linearly stable when $h^2\lambda_{\max}(M^{-1}K)<4$.

Implementation notes

1. Exact integration point: implement a PyTorch optimizer maintaining parameter tensors $\theta$, momentum tensors $p$, scalar time $t$, and scalar $e$. Begin with full-batch deterministic losses so that the canonical map can be tested without minibatch noise. Choose $M=mI$ or a diagonal mass estimated once from gradient statistics.

2. Pseudocode: initialize $p=0$, $t=0$, and $e=0$. For step size $h$, perform $t\leftarrow t+h/2$; then $\theta\leftarrow\theta+(h/2)M^{-1}p$; evaluate $g=\nabla_\theta L(\theta,t)$ and, if available, $\ell_t=\partial_tL(\theta,t)$; update $p\leftarrow p-hg$ and $e\leftarrow e-h\ell_t$; perform $\theta\leftarrow\theta+(h/2)M^{-1}p$; finally update $t\leftarrow t+h/2$. For a time-independent loss, set $\ell_t=0$. A practical implementation can omit $e$ from parameter updates but should retain it for energy diagnostics.

3. Derived versus estimated quantities: the Hamiltonian split, subflow updates, symplectic identity, and stability inequality are the structure-preserving construction. Estimate $\lambda_{\max}(M^{-1}K)$ using 5--10 Hessian-vector power iterations. Estimate the Jacobian symplectic residual with finite differences or automatic differentiation on a small model: $r=\|J^{\mathsf T}(D\Psi_h)^{\mathsf T}J(D\Psi_h)-J\|_F/\|J\|_F$. Avoid Adam-style damping in the first test because damping intentionally breaks volume preservation.

4. First cheap experiment: use a two-dimensional quadratic $L(\theta)=\frac12\theta^{\mathsf T}K\theta$ with known eigenvalues, followed by a two-layer MLP on MNIST. Compare the proposed optimizer against SGD with momentum and Adam while sweeping $h$ around $h_{\mathrm{crit}}=2/\sqrt{\lambda_{\max}(M^{-1}K)}$. The quantitative prediction is bounded oscillatory trajectories below the threshold and exponential divergence above it; the observed boundary should agree within 20%. The finite-step Jacobian should have determinant $1+O(10^{-5})$ in float64, and the symplectic residual should decrease approximately as $O(h^2)$ for the symmetric composition. On the MLP, measure long-horizon parameter norms, gradient norms, validation loss, and phase-space volume over 10,000 steps.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a PyTorch ExtendedSymplectic optimizer using the symmetric A-B-C-B-A split, retaining canonical momentum and extended time state, plus deterministic numerical tests. The toy mechanism manifested clearly: the predicted hcrit=1.0 was bracketed by 0.99/1.01, mass-scaled thresholds were observed at 0.475, 0.975, and 1.95 versus predictions 0.5, 1.0, and 2.0, and determinant/symplectic residuals were at 1e-16/zero roundoff. It did not beat practical baselines in the small logistic experiment: its final loss was 0.514 versus 0.2255 for momentum SGD and 0.2255 for Adam, with a much larger parameter norm.

Agent confidence: 9/10

Baseline: Momentum SGD final loss 0.2255095, Adam final loss 0.2255299; weight norms 2.4276 and 2.4285
Idea: Extended symplectic final loss 0.5140183, minimum loss 0.2262747, weight norm 10.3575; toy hcrit predicted 1.0 and observed bracket [0.99, 1.01], mass-scaling midpoint errors <=5%, determinant error <=2.2e-16

Limitations: Only a full-batch 2D quadratic and tiny logistic-regression task were tested; no MNIST/MLP, minibatch noise, long-horizon 10,000-step study, GPU benchmark, nontrivial time-dependent loss, or learned/diagonal mass matrix was evaluated. The optimizer currently assumes static losses for the energy conjugate and requires a closure.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.12914) Sep 2, 2026

Significantly WORSE than the tuned baseline (+652.2%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.6094
Idea mean: 0
Effect (Δ): 49.6313 (+652.2%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±98.5%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the structurally matched tabular track was tested; vision, sequence, and dynamics tracks were not tested. The benchmark used the fixed scalar mass and a 20-epoch small-data setting; no learned or diagonal mass, damping/restart variant, time-dependent loss, or 10,000-step long-horizon study was evaluated.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented the extended-symplectic optimizer as a local Verlet-style training loop on the structurally matched tabular Friedman#1 regression track using the shared mlp_tiny architecture. Across 8 paired seeds, tuned Adam achieved mean test MSE 7.6093 versus 57.2406 for the idea, with paired delta +49.6313 and permutation p=0.0081; the idea was significantly worse. The trained-model mechanism signature also failed: observed perturbation gain 452.81 versus predicted 1.0.

Baseline: Adam best config lr=0.01, weight_decay=0.0001; full 8-seed mean test MSE 7.609379350423813
Idea: Extended-symplectic best config lr=0.01, mass=1.0; full 8-seed mean test MSE 57.240631103515625
Idea wins: 0
Paired delta mean: 49.631251752376556
Permutation p value: 0.0081

Benchmark result