✗ Failed on benchmark 2026

IQC-Synthesized Momentum Optimizer

Implementation & benchmark of arXiv:2608.06915 — Stochastic Gradient Descent with Momentum: Analysis and Synthesis via Integral Quadratic Constraints

Usefulness7/10

Difficulty7/10

Novelty7/10

Optimization Speedup Stability Accuracy Optimizer Training dynamics Scheduler Control theory Optimization Linear algebra Stochastic processes Convex analysis

Source paper: Stochastic Gradient Descent with Momentum: Analysis and Synthesis via Integral Quadratic Constraints arXiv:2608.06915 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace hand-designed Heavy Ball or Nesterov coefficients with a low-order linear controller synthesized by a semidefinite program. The controller receives the stochastic mini-batch gradient and emits the parameter update; dynamic IQC multipliers constrain both gradient curvature and temporally correlated mini-batch noise, so the SDP directly minimizes a certified contraction factor rather than optimizing momentum heuristically.

Formulas

$$f(z)=\frac{1}{N}\sum_{i=1}^{N}f_i(z),\qquad \nabla f_{\mathscr I}(z)=\frac{1}{|\mathscr I|}\sum_{i\in\mathscr I}\nabla f_i(z),\qquad b=|\mathscr I|.$$

$$\bar q(\lambda)=\begin{pmatrix}\lambda_s\\\rho\lambda_s+\lambda_{s-1}\\\vdots\\\rho^{s-1}\lambda_s+\cdots+\lambda_1\end{pmatrix},\qquad q(\lambda)=\rho^s\lambda_s+\cdots+\rho\lambda_1+\lambda_0.$$

$$0<\rho^{2}\overline P_{22}-\overline P_{22}+\frac{2ml}{(l-m)^2}q(\lambda),$$

$$0\prec\begin{pmatrix}P_{22}&P_{23}&q(\lambda)\\P_{32}&P_{33}&-\frac{m}{l-m}\bar q(\lambda)\\q(\lambda)&-\frac{m}{l-m}\bar q(\lambda)&\overline P_{22}\end{pmatrix},\qquad \rho^{-2}P_{22}-2\tilde b\sum_{j=0}^{s}\rho^{-j}\lambda_j<0.$$

Mathematical statement

The objective is the composite finite-sum function $f(z)=\frac{1}{N}\sum_{i=1}^{N}f_i(z)$, with mini-batch gradient $\nabla f_{\mathscr I}(z)=\frac{1}{b}\sum_{i\in\mathscr I}\nabla f_i(z)$ for a sampled set $\mathscr I$ of size $b$. Let $m$ and $l$ denote lower and upper curvature bounds, typically strong-convexity and smoothness constants, with $0<m<l$. The optimizer is represented as a discrete controller with state $x_t^c$, input gradient $g_t$, and output update $u_t$: $x_{t+1}^c=A_cx_t^c+B_cg_t$ and $u_t=C_cx_t^c+D_cg_t$. The synthesis theorem searches for $A_c,B_c,C_c,D_c$, a positive-definite Lyapunov certificate $P$, scalar $\overline P_{22}$, and multiplier coefficients satisfying the LMIs (55a)-(55d). The dynamic multiplier is parameterized by $\lambda=(\lambda_0,\ldots,\lambda_s)$ and target rate $\rho$, with $\bar q(\lambda)=(\lambda_s,\rho\lambda_s+\lambda_{s-1},\ldots,\rho^{s-1}\lambda_s+\cdots+\lambda_1)^\top$ and $q(\lambda)=\rho^s\lambda_s+\cdots+\rho\lambda_1+\lambda_0$. The key scalar restrictions are $\lambda_j\leq0$ for $j=1,\ldots,s$, $\sum_{j=0}^{s}\rho^{-j}\lambda_j\geq0$, and $\rho^{-2}P_{22}-2\tilde b\sum_{j=0}^{s}\rho^{-j}\lambda_j<0$, where $\tilde b$ is the batch-sampling noise coefficient and $P_{22}$ is a certificate block. The contraction condition is $0<\rho^2\overline P_{22}-\overline P_{22}+\frac{2ml}{(l-m)^2}q(\lambda)$ together with the block positive-definiteness condition shown below. A feasible $\rho<1$ certifies geometric decay, so minimizing $\rho$ gives the synthesized optimizer's target rate.

Implementation notes

1. Integration point: implement the synthesized controller as a PyTorch optimizer operating on flattened parameter blocks. For each block, maintain controller state x of dimension r, read the current mini-batch gradient g_t, compute u_t=C_c x_t+D_c g_t, update theta_{t+1}=theta_t-u_t, and then update x_{t+1}=A_c x_t+B_c g_t. Start with r=1 or r=2 so the optimizer adds only a few scalar or small-matrix coefficients. 2. Offline synthesis pseudocode: collect T gradients and parameter displacements from a short proxy training run; estimate curvature bounds m and l using Hessian-vector power iterations or conservative quantiles of observed gradient-Lipschitz ratios; estimate the batch-noise coefficient tilde_b from the covariance of gradients produced by independent mini-batches. For s=1 or s=2, create decision variables rho, lambda_0,...,lambda_s, controller matrices A_c,B_c,C_c,D_c, certificate blocks P_22,P_23,P_33, and scalar overline_P_22. Minimize rho subject to the displayed sign constraints and the full LMI (55a), instantiated with the chosen dynamic multiplier realization. If the canonical state-space realization is initially inconvenient, use the equivalent MVP finite impulse response filter u_t=D_cg_t+C_cx_t with x_t=[g_{t-1},...,g_{t-s}] and impose the multiplier quadratic form directly on this history. 3. The paper supplies the IQC/LMI structure and certificate logic; m, l, tilde_b, and multiplier order s are estimated empirically. Reject solutions with rho>=1 or non-positive-definite certificates, and scale the synthesized output to match the baseline initial learning rate. 4. First experiment: train a 2-layer MLP and a small ResNet on CIFAR-10, comparing IQC momentum against tuned SGD with momentum, Nesterov, and AdamW at equal batch size and FLOPs. Measure loss versus optimizer steps, gradient-norm variance, divergence frequency across five seeds, and final validation accuracy. The target signal is faster loss descent or lower variance at equal compute, with no increase in catastrophic runs; a secondary test is whether one controller synthesized on the proxy model transfers to a wider model.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 5 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built an IQC-inspired first-order FIR controller with offline robust coefficient search over curvature and correlated-noise grids, plus a PyTorch optimizer interface. The three toy mechanism checks matched predictions: the scalar stability boundary was observed exactly at alpha*k=2, recurrence decay matched spectral-radius predictions within 0.45%, and AR(1) noise gain scaled exactly as 1/(1-ar). In the fair common-random-number quadratic benchmark, the synthesized controller had lower minimum loss but worse final averaged loss than the grid-tuned baseline, so the mechanism is validated but no general optimization win was observed.

Agent confidence: 7/10

Baseline: Grid-tuned heavy-ball reduced final-20-step quadratic loss to 0.00103317 (alpha=0.02957, beta=0); minimum loss 0.00047067.
Idea: Robust proxy synthesis returned alpha=0.2750, beta=0.05191, predicted rho=0.64444 and noise gain=4.0; final-20-step loss 0.00492493, minimum loss 0.00005876. Stability sweep matched predicted stable/unstable labels and radii; contraction relative errors were 0%, 0.22%, and 0.45%; correlated-noise gain relative errors were 0%.

Limitations: This is not a faithful implementation of the full paper SDP (55a)-(55d): the omitted psi realization and controller-extraction/passivity design were replaced by a low-order robust spectral-radius/noise-gain proxy. Only deterministic scalar recurrences and a small 4D quadratic were tested; no CIFAR-10, MLP/ResNet, AdamW/Nesterov comparison, multi-seed study, FLOP accounting, or transfer experiment was performed.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.06915) Sep 2, 2026

No significant difference vs. the tuned baseline (+131.0%, p=0.063, wins 0 / 5 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 5
Baseline mean: 11.3488
Idea mean: 25.1541
Effect (Δ): 14.8665 (+131.0%; negative = idea better)
Wins: 0 / 5 paired seeds
p-value: 0.063 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 6 configs

Limitations:

The full paper SDP (LMIs 55a-55d) was not implemented; this is the low-order robust FIR proxy. Vision, sequence, and dynamics tracks were not tested. The executed harness output had only five paired idea seeds, several NaN measurements, and therefore does not constitute a complete eight-seed protocol.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the IQC-inspired low-order FIR momentum optimizer and tested it on the structurally matched tabular Friedman regression task with the shared mlp_tiny architecture. It was substantially worse than tuned SGD with momentum: paired delta was +14.8665 MSE and there were no idea wins; the mechanism signature was also not confirmed because several idea runs produced NaNs. The executed report contains only 5 idea seeds, so this is an incomplete 8-seed protocol result and worked=false.

Baseline: SGD with momentum; best cfg lr=0.001, momentum=0.9; 8-seed test MSE mean=11.3488068581, std=1.8547895369.
Comparison: delta_mean=+14.8665159225 (idea minus baseline; lower is better), p_value=0.06295, idea_wins=0, n_pairs=5, verdict=no significant win.
Idea: IQC/FIR controller; best tested cfg lr=0.001; 5-seed test MSE mean=25.1540710449, std=1.4462706653. lr=0.003 diverged with sweep mean=8679688192.0.
Mechanism signature: Predicted update-variance reduction was not confirmed: idea variance mean=NaN, baseline variance mean=0.0140025184, observed ratio=NaN, confirmed=false.

Benchmark result