# Pair-monotone implicit optimizer

- ID: 1678
- Canonical URL: https://synthcore.org/idea/1678/pair-monotone-implicit-optimizer
- API JSON: https://synthcore.org/api/idea/1678.json
- API Markdown: https://synthcore.org/api/idea/1678.md
- Verification status: unverified
- Source: [arXiv:2608.01584](https://arxiv.org/abs/2608.01584)
- Category: optimization
- Solves: stability, accuracy
- ML areas: optimizer, training-dynamics, mlp
- Math tags: optimization, convex-analysis, numerical-analysis, dynamical-systems
- Ratings: usefulness 5/10; difficulty 7/10; novelty 6/10

## Idea description

Replace the explicit parameter update \(\theta_{k+1}=\theta_k-\eta\nabla L(\theta_k)\) with an approximate generalized proximal step defined by a simple map \(v\). The map is chosen so that the gradient operator and v satisfy an empirical pair-monotonicity condition, allowing larger stable outer steps and reducing oscillations in stiff or highly curved neural-network training.

## Mathematical statement

The paper formulates the inclusion problem \(0\in F(x)\) and recalls the proximal iteration \(x_{k+1}=J_{\gamma F}(x_k)\), where \(J_{\gamma F}=(Id+\gamma F)^{-1}\); equivalently, \((x_{k+1}-x_k)/\gamma\in-F(x_{k+1})\). Its key generalized structure is a pair \((F,v)\), with \(F:\mathcal H\rightrightarrows\mathcal H\) and \(v:\mathcal H\to\mathcal H\), satisfying \(\langle F(x)-F(y),v(x)-v(y)\rangle\geq\alpha\|x-y\|^2\) for some \(\alpha>0\). Here \(\mathcal H\) is the parameter Euclidean space, \(F(\theta)=\nabla L(\theta)\), \(v(\theta)=P\theta\), and P is a positive diagonal preconditioner. The generalized implicit step is implemented as \((v(\theta_{k+1})-v(\theta_k))/\gamma\approx-F(\theta_{k+1})\), or, for linear v, \(P(\theta_{k+1}-\theta_k)/\gamma\approx-\nabla L(\theta_{k+1})\). Pair monotonicity is estimated on minibatch perturbations rather than assumed globally: \(\hat\alpha=\min_i \langle g_i-g_i',P(\theta_i-\theta_i')\rangle/(\|\theta_i-\theta_i'\|^2+\varepsilon)\).

## Key formulas

- $$0\in F(x),$$
- $$x_{k+1}=J_{\gamma F}(x_k),\qquad J_{\gamma F}:=(Id+\gamma F)^{-1},$$
- $$\frac{x_{k+1}-x_k}{\gamma}\in-F(x_{k+1}),$$
- $$\langle F(x)-F(y),v(x)-v(y)\rangle\geq\alpha\|x-y\|^{2},\qquad \alpha>0.$$

## Implementation notes

Integrate this into the optimizer update for a small MLP or Transformer block, operating on the flattened parameter vector \(\theta\) or on one parameter group at a time. Use \(F(\theta)=\nabla_\theta L(\theta;B)\), and choose \(v(\theta)=P\theta\), where P is diagonal and initialized from Adam's second moment as \(P_j=(\sqrt{s_j}+\epsilon)^{-1}\), then frozen or slowly updated. The target step is the generalized proximal equation \(P(\theta^+-\theta)/\gamma+\nabla L(\theta^+;B)=0\). Pseudocode: `theta_old=theta; z=theta_old; for t in 1..T: loss=L(z,B); g=grad(loss,z); r=P*(z-theta_old)/gamma+g; z=z-lr_inner*r/(diag(P)/gamma+curvature_floor); theta=z`. Use one or two fixed-point or Newton-like inner iterations; detach P but retain gradients through z. Apply weight decay outside this equation. At every outer step compute the residual \(\|P(\theta-\theta_{old})/\gamma+\nabla L(\theta;B)\|\), the explicit baseline update norm, and the minibatch pair-monotonicity estimator \(\hat\alpha\): draw a small perturbation \(\delta\), evaluate gradients g at \(\theta\) and g' at \(\theta+\delta\), and set \(\hat\alpha=\langle g-g',P\delta\rangle/(\|\delta\|^2+\epsilon)\). The mathematics supplies the implicit equation and pair inequality; curvature, \(\hat\alpha\), and the acceptable inner residual are estimated empirically. First experiment: train a 2-4 layer CIFAR-10 MLP with SGD, AdamW, and this method at matched gradient-evaluation and FLOP budgets. Sweep \(\gamma\) from the AdamW learning rate to 10 times it. Measure training-loss oscillation, gradient-norm spikes, final accuracy, and loss decrease per outer step. Success means stable training at a substantially larger \(\gamma\), fewer divergence events, or higher accuracy at equal gradient evaluations; reject the method if inner solves erase any stability or convergence benefit.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.