# IQC-Synthesized Momentum Optimizer

- ID: 1915
- Canonical URL: https://synthcore.org/idea/1915/iqc-synthesized-momentum-optimizer
- API JSON: https://synthcore.org/api/idea/1915.json
- API Markdown: https://synthcore.org/api/idea/1915.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.06915](https://arxiv.org/abs/2608.06915)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: control-theory, optimization, linear-algebra, stochastic-processes, convex-analysis
- Ratings: usefulness 7/10; difficulty 7/10; novelty 7/10

## Idea description

Replace hand-designed Heavy Ball or Nesterov coefficients with a low-order linear controller synthesized by a semidefinite program. The controller receives the stochastic mini-batch gradient and emits the parameter update; dynamic IQC multipliers constrain both gradient curvature and temporally correlated mini-batch noise, so the SDP directly minimizes a certified contraction factor rather than optimizing momentum heuristically.

## Mathematical statement

The objective is the composite finite-sum function $f(z)=\frac{1}{N}\sum_{i=1}^{N}f_i(z)$, with mini-batch gradient $\nabla f_{\mathscr I}(z)=\frac{1}{b}\sum_{i\in\mathscr I}\nabla f_i(z)$ for a sampled set $\mathscr I$ of size $b$. Let $m$ and $l$ denote lower and upper curvature bounds, typically strong-convexity and smoothness constants, with $0<m<l$. The optimizer is represented as a discrete controller with state $x_t^c$, input gradient $g_t$, and output update $u_t$: $x_{t+1}^c=A_cx_t^c+B_cg_t$ and $u_t=C_cx_t^c+D_cg_t$. The synthesis theorem searches for $A_c,B_c,C_c,D_c$, a positive-definite Lyapunov certificate $P$, scalar $\overline P_{22}$, and multiplier coefficients satisfying the LMIs (55a)-(55d). The dynamic multiplier is parameterized by $\lambda=(\lambda_0,\ldots,\lambda_s)$ and target rate $\rho$, with $\bar q(\lambda)=(\lambda_s,\rho\lambda_s+\lambda_{s-1},\ldots,\rho^{s-1}\lambda_s+\cdots+\lambda_1)^\top$ and $q(\lambda)=\rho^s\lambda_s+\cdots+\rho\lambda_1+\lambda_0$. The key scalar restrictions are $\lambda_j\leq0$ for $j=1,\ldots,s$, $\sum_{j=0}^{s}\rho^{-j}\lambda_j\geq0$, and $\rho^{-2}P_{22}-2\tilde b\sum_{j=0}^{s}\rho^{-j}\lambda_j<0$, where $\tilde b$ is the batch-sampling noise coefficient and $P_{22}$ is a certificate block. The contraction condition is $0<\rho^2\overline P_{22}-\overline P_{22}+\frac{2ml}{(l-m)^2}q(\lambda)$ together with the block positive-definiteness condition shown below. A feasible $\rho<1$ certifies geometric decay, so minimizing $\rho$ gives the synthesized optimizer's target rate.

## Key formulas

- $$f(z)=\frac{1}{N}\sum_{i=1}^{N}f_i(z),\qquad \nabla f_{\mathscr I}(z)=\frac{1}{|\mathscr I|}\sum_{i\in\mathscr I}\nabla f_i(z),\qquad b=|\mathscr I|.$$
- $$\bar q(\lambda)=\begin{pmatrix}\lambda_s\\\rho\lambda_s+\lambda_{s-1}\\\vdots\\\rho^{s-1}\lambda_s+\cdots+\lambda_1\end{pmatrix},\qquad q(\lambda)=\rho^s\lambda_s+\cdots+\rho\lambda_1+\lambda_0.$$
- $$0<\rho^{2}\overline P_{22}-\overline P_{22}+\frac{2ml}{(l-m)^2}q(\lambda),$$
- $$0\prec\begin{pmatrix}P_{22}&P_{23}&q(\lambda)\\P_{32}&P_{33}&-\frac{m}{l-m}\bar q(\lambda)\\q(\lambda)&-\frac{m}{l-m}\bar q(\lambda)&\overline P_{22}\end{pmatrix},\qquad \rho^{-2}P_{22}-2\tilde b\sum_{j=0}^{s}\rho^{-j}\lambda_j<0.$$

## Implementation notes

1. Integration point: implement the synthesized controller as a PyTorch optimizer operating on flattened parameter blocks. For each block, maintain controller state x of dimension r, read the current mini-batch gradient g_t, compute u_t=C_c x_t+D_c g_t, update theta_{t+1}=theta_t-u_t, and then update x_{t+1}=A_c x_t+B_c g_t. Start with r=1 or r=2 so the optimizer adds only a few scalar or small-matrix coefficients. 2. Offline synthesis pseudocode: collect T gradients and parameter displacements from a short proxy training run; estimate curvature bounds m and l using Hessian-vector power iterations or conservative quantiles of observed gradient-Lipschitz ratios; estimate the batch-noise coefficient tilde_b from the covariance of gradients produced by independent mini-batches. For s=1 or s=2, create decision variables rho, lambda_0,...,lambda_s, controller matrices A_c,B_c,C_c,D_c, certificate blocks P_22,P_23,P_33, and scalar overline_P_22. Minimize rho subject to the displayed sign constraints and the full LMI (55a), instantiated with the chosen dynamic multiplier realization. If the canonical state-space realization is initially inconvenient, use the equivalent MVP finite impulse response filter u_t=D_cg_t+C_cx_t with x_t=[g_{t-1},...,g_{t-s}] and impose the multiplier quadratic form directly on this history. 3. The paper supplies the IQC/LMI structure and certificate logic; m, l, tilde_b, and multiplier order s are estimated empirically. Reject solutions with rho>=1 or non-positive-definite certificates, and scale the synthesized output to match the baseline initial learning rate. 4. First experiment: train a 2-layer MLP and a small ResNet on CIFAR-10, comparing IQC momentum against tuned SGD with momentum, Nesterov, and AdamW at equal batch size and FLOPs. Measure loss versus optimizer steps, gradient-norm variance, divergence frequency across five seeds, and final validation accuracy. The target signal is faster loss descent or lower variance at equal compute, with no increase in catastrophic runs; a secondary test is whether one controller synthesized on the proxy model transfers to a wider model.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built an IQC-inspired first-order FIR controller with offline robust coefficient search over curvature and correlated-noise grids, plus a PyTorch optimizer interface. The three toy mechanism checks matched predictions: the scalar stability boundary was observed exactly at alpha*k=2, recurrence decay matched spectral-radius predictions within 0.45%, and AR(1) noise gain scaled exactly as 1/(1-ar). In the fair common-random-number quadratic benchmark, the synthesized controller had lower minimum loss but worse final averaged loss than the grid-tuned baseline, so the mechanism is validated but no general optimization win was observed.

### Mechanism check

- Verdict: Built an IQC-inspired first-order FIR controller with offline robust coefficient search over curvature and correlated-noise grids, plus a PyTorch optimizer interface. The three toy mechanism checks matched predictions: the scalar stability boundary was observed exactly at alpha*k=2, recurrence decay matched spectral-radius predictions within 0.45%, and AR(1) noise gain scaled exactly as 1/(1-ar). In the fair common-random-number quadratic benchmark, the synthesized controller had lower minimum loss but worse final averaged loss than the grid-tuned baseline, so the mechanism is validated but no general optimization win was observed.
- Confidence: 7/10
- Limitations: This is not a faithful implementation of the full paper SDP (55a)-(55d): the omitted psi realization and controller-extraction/passivity design were replaced by a low-order robust spectral-radius/noise-gain proxy. Only deterministic scalar recurrences and a small 4D quadratic were tested; no CIFAR-10, MLP/ResNet, AdamW/Nesterov comparison, multi-seed study, FLOP accounting, or transfer experiment was performed.

### Practical benchmark

- Paired seeds: 5
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.06295
- Paired wins: 0/5
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/728/bench_report.json)
- [iqc_optimizer.py](https://synthcore.org/code/728/iqc_optimizer.py)
- [report.md](https://synthcore.org/code/728/report.md)
- [report_bench_2026-09-01T013202.md](https://synthcore.org/code/728/report_bench_2026-09-01T013202.md)
- [results.json](https://synthcore.org/code/728/results.json)
- [run_experiment.py](https://synthcore.org/code/728/run_experiment.py)
- [stage2_bench.py](https://synthcore.org/code/728/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/728)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.