# Warped nonmonotone proximal optimizer

- ID: 71
- Canonical URL: https://synthcore.org/idea/71/warped-nonmonotone-proximal-optimizer
- API JSON: https://synthcore.org/api/idea/71.json
- API Markdown: https://synthcore.org/api/idea/71.md
- Verification status: unverified
- Source: [arXiv:2608.22687](https://arxiv.org/abs/2608.22687)
- Category: optimization
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics
- Math tags: optimization, convex-analysis, dynamical-systems, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace the usual Euclidean gradient step by a warped resolvent step with a learned diagonal metric, and explicitly account for the fact that minibatch or approximate gradients are non-monotone. The metric and step scale are increased when an empirical semimonotonicity test detects negative curvature or gradient mismatch, producing a practical stability safeguard without requiring the neural loss itself to be convex.

## Mathematical statement

The paper studies inclusions of the form $0\in(A+C+D)x+L^{*}u$ and, in the primal-only case, $0\in(A+L^{*}BL+C+D)x$, where $A$ is a set-valued proximal operator, $C$ is $\beta$-cocoercive, and $D$ is $\vartheta$-Lipschitz but may be non-monotone. Cocoercivity means $\langle Cx-Cy,x-y\rangle\geq\beta\|Cx-Cy\|^{2}$ for $\beta>0$; Lipschitzness means $\|Dx-Dy\|\leq\vartheta\|x-y\|$. The warped resolvent uses an invertible operator $\bm M$ and the update $z=(\bm M+\bm A)^{-1}(\bm Mx-Fx)$, with $F=C+D$. The extracted convergence assumption introduces a nonexpansive map $\bm S$, a scale $\tau$, and $\zeta\in[0,1/2)$ such that $\tau\bm M-\bm S$ is $\zeta$-Lipschitz relative to $\bm S$. Its explicit admissible lower bound is $\tau>\underline{\tau}_{\zeta}$, where $\hat\rho=\min\{\rho,0\}$ and $\underline{\tau}_{\zeta}=-\zeta\hat\rho/(\beta+\hat\rho)$. Here $\rho$ is the semimonotonicity modulus, meaning $\langle Fx-Fy,x-y\rangle\geq\rho\|x-y\|^{2}$; negative $\rho$ quantifies non-monotonicity. For optimization, take $A=\partial R$ for a regularizer $R$, $C$ as a stable or cocoercive gradient component, $D$ as the non-monotone residual, $S=I$, and $M$ as a positive diagonal preconditioner.

## Key formulas

- $$\text{find }x\in\mathcal{H}\text{ such that }0\in(A+L^{*}BL+C+D)x.$$
- $$z=(\bm M+\bm A)^{-1}(\bm Mx-Fx),\qquad F=C+D.$$
- $$\hat{\rho}=\min\{\rho,0\},\qquad \underline{\tau}_{\zeta}=-\frac{\zeta\hat{\rho}}{\beta+\hat{\rho}},\qquad \tau>\underline{\tau}_{\zeta}.$$
- $$\langle Fx-Fy,x-y\rangle\geq\rho\|x-y\|^{2},\qquad \|Dx-Dy\|\leq\vartheta\|x-y\|.$$

## Implementation notes

(1) Integration point: replace the AdamW or SGD parameter update during pretraining or fine-tuning. Partition the minibatch gradient into a stable component $c_t$ and residual component $d_t$; the simplest MVP uses $c_t=0$ and treats the full gradient as $d_t$, while a stronger version uses a moving-average gradient as $c_t$ and the minibatch-minus-moving-average residual as $d_t$. Store a positive diagonal metric $M_t$ (one scalar per parameter tensor or one EMA second moment per parameter). Set $A=\partial R$, where $R$ is weight decay or a proximal constraint. (2) Pseudocode: compute $g_t=C_t+D_t$; draw or retain the previous minibatch gradient $g_{t-1}$; estimate $\rho_t=\langle g_t-g_{t-1},w_t-w_{t-1}\rangle/(\|w_t-w_{t-1}\|^2+\epsilon)$ and use $\hat\rho_t=\min(\rho_t,0)$. Estimate $\beta_t$ from the stable component with $\beta_t=\|c_t-c_{t-1}\|^2/(\langle c_t-c_{t-1},w_t-w_{t-1}\rangle+\epsilon)$, clipped to a safe interval. Choose $\zeta=0.25$, $\tau_t=1.1[-\zeta\hat\rho_t/(\beta_t+\hat\rho_t)]_+$, plus a fixed positive floor. Form $v_t=w_t-M_t^{-1}g_t/\tau_t$ and apply the metric proximal map $w_{t+1}=\operatorname{prox}^{M_t}_{R}(v_t)$, where $\operatorname{prox}^{M}_{R}(v)=\arg\min_w R(w)+\frac12\|w-v\|_M^2$. Backtrack by multiplying $\tau_t$ by two if the observed secant inequality violates the chosen bound. (3) The paper supplies the warped-resolvent structure and explicit dependence on negative semimonotonicity; the secant estimates of $\rho_t$, $\beta_t$, and the diagonal metric are empirical engineering approximations. (4) First experiment: train a 6-layer MLP or small ViT on CIFAR-10, comparing the method with AdamW at equal parameter count, minibatch size, and FLOPs. Measure loss decrease per optimizer step, gradient-norm spikes, divergence frequency over five seeds, and final validation accuracy. The first success signal is fewer exploding-loss runs and faster loss descent at the same compute; a secondary signal is improved accuracy under deliberately noisy or stale gradients.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.