# Strongly Pseudomonotone Implicit Router

- ID: 1176
- Canonical URL: https://synthcore.org/idea/1176/strongly-pseudomonotone-implicit-router
- API JSON: https://synthcore.org/api/idea/1176.json
- API Markdown: https://synthcore.org/api/idea/1176.md
- Verification status: unverified
- Source: [arXiv:2607.20133](https://arxiv.org/abs/2607.20133)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: moe-routing, architecture, regularization
- Math tags: convex-analysis, optimization, dynamical-systems
- Ratings: usefulness 6/10; difficulty 6/10; novelty 5/10

## Idea description

Replace an explicit MoE router or constrained output head with the solution of a variational inequality over a convex feasible set. The neural operator can be nonmonotone, but training should enforce a measurable strong-pseudomonotonicity margin so the selected route or control is unique and has bounded sensitivity to changes in the token representation. Use an explicit projection residual for approximate solving and for monitoring whether the implicit layer has actually converged.

## Mathematical statement

For a nonempty closed convex set $C\subset H$, the projection is characterized by $y=\operatorname{proj}_C(z)$ if and only if $y\in C$ and $\langle z-y,v-y\rangle\le 0$ for every $v\in C$. The implicit response $u(x)$ is defined by the variational inequality $\langle F(x,u(x)),v-u(x)\rangle\ge 0$ for every $v\in C$, where $x$ is the neural input, $u\in C$ is the output, and $F$ is a learned operator. Use the paper's strong-pseudomonotonicity condition: for all $u,v\in C$, $\langle F(x,u),v-u\rangle\ge 0$ implies $\langle F(x,v),v-u\rangle\ge \mu\|v-u\|^2$, with margin $\mu>0$. Assume $F$ is state-Lipschitz, $\|F(x,u)-F(y,u)\|\le L_x\|x-y\|$, and use the sensitivity target $\|u(x)-u(y)\|\lesssim (L_x/\mu)\|x-y\|$. The projection residual from the paper is $R_\eta(x,u)=u-\operatorname{proj}_C(u-\eta F(x,u))$; an exact VI solution has $R_\eta(x,u)=0$ for any $\eta>0$.

## Key formulas

- $$y=\operatorname{proj}_{C}(z)\iff\left(y\in C\ \text{and}\ \langle z-y,v-y\rangle\leq 0,\ \forall v\in C\right).$$
- $$\langle F(x,u),v-u\rangle\geq 0,\quad \forall v\in C,$$
- $$\langle F(x,u),v-u\rangle\geq 0\ \Longrightarrow\ \langle F(x,v),v-u\rangle\geq \mu\|v-u\|^2,$$
- $$R_\eta(x,u):=u-\operatorname{proj}_{C}\bigl(u-\eta F(x,u)\bigr),\qquad \|R_\eta(x,u)\|\to 0.$$

## Implementation notes

Integrate the layer immediately before an MoE dispatch decision or constrained prediction head. For each token representation $x_i\in\mathbb{R}^d$, let the neural operator output $F_\theta(x_i,u)=A_\theta(x_i)u+b_\theta(x_i)$, where $u\in\mathbb{R}^m$ is a continuous router vector; choose $C$ as the probability simplex for mixture weights, a box, or a capped simplex enforcing a maximum expert load. Solve the VI with projected extragradient: initialize $u_0=\operatorname{proj}_C(-F_\theta(x,0))$; for $k=0,\ldots,K-1$, compute $z_k=\operatorname{proj}_C(u_k-\eta F_\theta(x,u_k))$ and $u_{k+1}=\operatorname{proj}_C(u_k-\eta F_\theta(x,z_k))$. Dispatch using $u_K$ and optionally use straight-through top-1 or top-2 selection. Add the residual loss $\lambda_R\|R_\eta(x,u_K)\|^2$ and a sensitivity regularizer on sampled pairs $(x,y)$, penalizing $\max(0,\|u_K(x)-u_K(y)\|-\kappa\|x-y\|)^2$, with $\kappa$ intended to approximate $L_x/\mu$. Estimate $L_x$ by minibatch finite differences and estimate a local strong-pseudomonotonicity margin by sampling $u,v\in C$ and measuring $\langle F(x,v),v-u\rangle/\|v-u\|^2$ whenever $\langle F(x,u),v-u\rangle\ge0$; reject or clip batches whose estimated margin is below a small positive threshold. The mathematical quantities are the VI, projection residual, and target sensitivity ratio; the Lipschitz and pseudomonotonicity constants are empirical diagnostics rather than certified global values. First test on a four-expert Switch-Transformer-style router on CIFAR-100 or a small language-model classification task, comparing against a standard softmax router at equal total FLOPs. Use $K=2,4,8$ inner iterations and report validation loss, expert-load variance, residual norm, gradient-norm spikes, and robustness under Gaussian token perturbations. Success means lower load collapse and fewer instability spikes at comparable perplexity or accuracy; a stronger result is improved perturbed accuracy without increasing the number of dispatched experts.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.