Warped nonmonotone proximal optimizer
Implementation & benchmark of arXiv:2608.22687 — Nonlinear Forward-Backward Algorithm for Solving Non-monotone+Lipschitz Inclusions with Applications to Adjoint Mismatch Problems
Source paper: Nonlinear Forward-Backward Algorithm for Solving Non-monotone+Lipschitz Inclusions with Applications to Adjoint Mismatch Problems arXiv:2608.22687 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace the usual Euclidean gradient step by a warped resolvent step with a learned diagonal metric, and explicitly account for the fact that minibatch or approximate gradients are non-monotone. The metric and step scale are increased when an empirical semimonotonicity test detects negative curvature or gradient mismatch, producing a practical stability safeguard without requiring the neural loss itself to be convex.
Formulas
Mathematical statement
The paper studies inclusions of the form $0\in(A+C+D)x+L^{*}u$ and, in the primal-only case, $0\in(A+L^{*}BL+C+D)x$, where $A$ is a set-valued proximal operator, $C$ is $\beta$-cocoercive, and $D$ is $\vartheta$-Lipschitz but may be non-monotone. Cocoercivity means $\langle Cx-Cy,x-y\rangle\geq\beta\|Cx-Cy\|^{2}$ for $\beta>0$; Lipschitzness means $\|Dx-Dy\|\leq\vartheta\|x-y\|$. The warped resolvent uses an invertible operator $\bm M$ and the update $z=(\bm M+\bm A)^{-1}(\bm Mx-Fx)$, with $F=C+D$. The extracted convergence assumption introduces a nonexpansive map $\bm S$, a scale $\tau$, and $\zeta\in[0,1/2)$ such that $\tau\bm M-\bm S$ is $\zeta$-Lipschitz relative to $\bm S$. Its explicit admissible lower bound is $\tau>\underline{\tau}_{\zeta}$, where $\hat\rho=\min\{\rho,0\}$ and $\underline{\tau}_{\zeta}=-\zeta\hat\rho/(\beta+\hat\rho)$. Here $\rho$ is the semimonotonicity modulus, meaning $\langle Fx-Fy,x-y\rangle\geq\rho\|x-y\|^{2}$; negative $\rho$ quantifies non-monotonicity. For optimization, take $A=\partial R$ for a regularizer $R$, $C$ as a stable or cocoercive gradient component, $D$ as the non-monotone residual, $S=I$, and $M$ as a positive diagonal preconditioner.
Implementation notes
(1) Integration point: replace the AdamW or SGD parameter update during pretraining or fine-tuning. Partition the minibatch gradient into a stable component $c_t$ and residual component $d_t$; the simplest MVP uses $c_t=0$ and treats the full gradient as $d_t$, while a stronger version uses a moving-average gradient as $c_t$ and the minibatch-minus-moving-average residual as $d_t$. Store a positive diagonal metric $M_t$ (one scalar per parameter tensor or one EMA second moment per parameter). Set $A=\partial R$, where $R$ is weight decay or a proximal constraint. (2) Pseudocode: compute $g_t=C_t+D_t$; draw or retain the previous minibatch gradient $g_{t-1}$; estimate $\rho_t=\langle g_t-g_{t-1},w_t-w_{t-1}\rangle/(\|w_t-w_{t-1}\|^2+\epsilon)$ and use $\hat\rho_t=\min(\rho_t,0)$. Estimate $\beta_t$ from the stable component with $\beta_t=\|c_t-c_{t-1}\|^2/(\langle c_t-c_{t-1},w_t-w_{t-1}\rangle+\epsilon)$, clipped to a safe interval. Choose $\zeta=0.25$, $\tau_t=1.1[-\zeta\hat\rho_t/(\beta_t+\hat\rho_t)]_+$, plus a fixed positive floor. Form $v_t=w_t-M_t^{-1}g_t/\tau_t$ and apply the metric proximal map $w_{t+1}=\operatorname{prox}^{M_t}_{R}(v_t)$, where $\operatorname{prox}^{M}_{R}(v)=\arg\min_w R(w)+\frac12\|w-v\|_M^2$. Backtrack by multiplying $\tau_t$ by two if the observed secant inequality violates the chosen bound. (3) The paper supplies the warped-resolvent structure and explicit dependence on negative semimonotonicity; the secant estimates of $\rho_t$, $\beta_t$, and the diagonal metric are empirical engineering approximations. (4) First experiment: train a 6-layer MLP or small ViT on CIFAR-10, comparing the method with AdamW at equal parameter count, minibatch size, and FLOPs. Measure loss decrease per optimizer step, gradient-norm spikes, divergence frequency over five seeds, and final validation accuracy. The first success signal is fewer exploding-loss runs and faster loss descent at the same compute; a secondary signal is improved accuracy under deliberately noisy or stale gradients.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.