✗ Failed on benchmark 2026

Order-Adaptive Integral Optimizer

Implementation & benchmark of arXiv:2609.00688 — Order-Adaptive Distributed Integral Control

Usefulness8/10

Difficulty5/10

Novelty7/10

Dynamics Stability Speedup Sample efficiency Optimizer Training dynamics Scheduler Control theory Dynamical systems Linear algebra Numerical analysis

Source paper: Order-Adaptive Distributed Integral Control arXiv:2609.00688 ⓘ · analyzed Sep 2, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a fixed optimizer memory order with a nested family of gradient-integral controllers. Training begins with a first-order update and activates additional accumulated-gradient states only after an exponentially smoothed residual fails to decrease for several decision intervals; newly activated gains are ramped from zero, so the parameter update remains continuous and previously learned states are preserved. The optimizer should use little memory on easy problems and acquire longer memory on ill-conditioned or delayed-gradient problems.

Formulas

$$\dot z_1(t)=g(\theta(t)),\qquad \dot z_{r+1}(t)=z_r(t),\quad r=1,\ldots,p-1$$

$$\dot\theta(t)=-a_0g(\theta(t))-\sum_{r=1}^{p}a_rz_r(t),\qquad P_{p,q}(\lambda)=\lambda^{p+1}+a_0q\lambda^p+\sum_{r=1}^{p}a_rq\lambda^{p-r}$$

$$a_l^2>4a_{l-1}a_{l+1}\quad(l=1,\ldots,p-1),\qquad a_l>0$$

$$r_k=\beta r_{k-1}+(1-\beta)\|g_k\|_2,\qquad p_{k+1}=p_k+1\ \text{if}\ r_{k-j}>\rho r_{k-j-1}\ \forall j=0,\ldots,K-1$$

Mathematical statement

For parameters $\theta\in\mathbb{R}^d$, gradient $g(\theta)=\nabla L(\theta)$, and order $p$, maintain integral-chain states $z_1,\ldots,z_p\in\mathbb{R}^d$ with $\dot z_1=g$ and $\dot z_{r+1}=z_r$. The continuous optimizer is $\dot\theta=-a_0g-\sum_{r=1}^{p}a_rz_r$. Around a quadratic mode $L(\theta)=\frac12q\theta^2$, where $q>0$, each eigenmode has characteristic polynomial $P_{p,q}(\lambda)=\lambda^{p+1}+a_0q\lambda^p+a_1q\lambda^{p-1}+\cdots+a_pq$. Choose positive coefficients satisfying the paper's adjacent log-concavity condition $a_l^2>4a_{l-1}a_{l+1}$; this is a conservative design rule for separated, nonoscillatory roots. Candidate orders are nested because order $p+1$ retains $z_1,\ldots,z_p$ and adds only $z_{p+1}$. Let $r_k$ be an exponential moving average of gradient norm or prediction residual. If $r_k$ fails to contract below a threshold for $K$ decision intervals, increase $p$. A smooth activation $\gamma_p(t)\in[0,1]$ multiplies the newly added gain and prevents discontinuous updates.

Implementation notes

1. Integration point: implement this as a PyTorch optimizer around any MLP, CNN, Transformer, or language-model training loop. Start with order $p=0$, meaning ordinary gradient descent or AdamW without changing the network architecture. For each parameter tensor, maintain optional states $z_1,z_2,\ldots,z_{p_{\max}}$; use a global order decision initially, then test layerwise order as a follow-up. 2. Pseudocode: compute gradients $g$; update $r\leftarrow\beta r+(1-\beta)\|g\|$; if the contraction test fails for $K$ checks, increment $p$; initialize the new state $z_{p+1}=0$, set its activation $\gamma=0$, and ramp $\gamma\leftarrow\min(1,\gamma+1/H)$ over $H$ optimizer steps; update $z_1\leftarrow z_1+\eta g$, $z_{r+1}\leftarrow z_{r+1}+\eta z_r$, and $\theta\leftarrow\theta-\eta[a_0g+\sum_r\gamma_ra_rz_r]$. Apply norm clipping to every state to prevent integral windup. 3. Computed from the paper: nested order expansion, residual-triggered finite decision intervals, continuous transitions, and polynomial gain constraints. Estimated empirically: gradient contraction $r_k/r_{k-1}$, Hessian curvature range, and the discrete-time stability boundary. For a local linear model, estimate dominant Jacobian eigenvalues by Lanczos and compare the measured Euler stability limit $\eta_c$ with $\eta_c=\min_j[-2\operatorname{Re}(\lambda_j)/|\lambda_j|^2]$. 4. First cheap experiment: train a 2-layer MLP on MNIST and a small Transformer on Penn Treebank, comparing SGD, fixed momentum, fixed order-1 integral memory, and adaptive orders $0\rightarrow1\rightarrow2$. Use $\beta=0.95$, $\rho=0.98$, $K=20$, $H=50$, and coefficients satisfying the displayed inequality. The primary falsifiable signature is a plateau in $r_k$ followed by an order transition, after which residual decay accelerates without a discontinuous loss spike. On quadratic regression, the measured divergence boundary should agree with the Jacobian prediction within 20%; on easy well-conditioned data, the adaptive method should remain at $p=0$ for most training, while on ill-conditioned data it should activate higher order and produce faster late-time residual decay.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a PyTorch order-adaptive integral optimizer with lazy nested states, EMA residual-triggered order growth, gain ramping, and state clipping. The quadratic check verified the coefficient inequality and stable continuous-time polynomial roots; on slow quadratics, order transitions reproducibly occurred and reduced final loss versus SGD, for example 0.0002374 versus 0.0002991 on curvature [1, 0.01]. The effect was absent on easy and rapidly solved cases, and the MLP smoke test matched SGD, so this is a promising toy signal rather than a demonstrated general training win.

Agent confidence: 7/10

Baseline: Slow quadratic [1, 0.01], 2000 steps: final loss 2.9912e-4; [1, 0.1]: 8.1569e-5. Easy [1,2,4], 1200 steps: 2.5314e-17.
Idea: Slow quadratic [1, 0.01], 2000 steps: final loss 2.3742e-4 with p=1 at step 30 and p=2 at step 50; [1, 0.1]: 6.7628e-5. Easy [1,2,4]: unchanged at 2.5314e-17 and remained p=0. MLP smoke test: identical to SGD, final loss 0.04702.

Limitations: The mini-experiment used deterministic quadratic systems and one tiny synthetic MLP only; MNIST, Penn Treebank, delayed stochastic gradients, fixed higher-order baselines, wall-clock speed, memory overhead, and broad seed sweeps were not tested. The slow-quadratic probe was run interactively and its summary is not separately stored in results.json.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.00688) Sep 2, 2026

Significantly WORSE than the tuned baseline (+36.7%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 8.4621
Idea mean: 11.5672
Effect (Δ): 3.1051 (+36.7%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the structurally matched built-in tabular/Friedman regression track with mlp_tiny was tested; vision, sequence, and dynamics tracks were not tested. The run used 400 training samples, 400 test samples, 18 epochs, and did not measure wall-clock speed or memory overhead. The baseline sweep used SGD momentum and three learning rates; no AdamW or layerwise-order variant was evaluated.

How to run: PYTHONPATH=/home/maxwelhelp/all/math2nn /home/maxwelhelp/main/bin/python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 10/10

Built a local Order-Adaptive Integral optimizer with nested states, EMA-triggered order growth, gain ramping, and clipping, evaluated against the same tabular MLP using the bench protocol. Across 8 paired seeds, tuned momentum SGD achieved test MSE 8.4621 while the best adaptive setting achieved 11.5672; the paired permutation p-value was 0.0081 for a significantly worse result. The mechanism signature was not confirmed because order activated on all seeds rather than remaining mostly at order zero on this easy tabular task.

Baseline: Tuned SGD with momentum=0.9, lr=0.003; 8-seed test MSE mean 8.4620805383, std 0.7635960927.
Idea: Order-adaptive integral optimizer, best tested lr=0.003; 8-seed test MSE mean 11.5672200918, std 1.9770478863. Paired delta +3.1051395535, 0/8 wins, permutation p=0.0081; verdict idea worse (significant).

Benchmark result