# Forward-Invariant Expert Authority Router

- ID: 2397
- Canonical URL: https://synthcore.org/idea/2397/forward-invariant-expert-authority-router
- API JSON: https://synthcore.org/api/idea/2397.json
- API Markdown: https://synthcore.org/api/idea/2397.md
- Verification status: unverified
- Source: [arXiv:2608.16335](https://arxiv.org/abs/2608.16335)
- Category: regularization
- Solves: generalization, stability, accuracy
- ML areas: moe-routing, regularization, transformer, training-dynamics
- Math tags: linear-algebra, optimization, control-theory, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace unconstrained or entropy-regularized MoE routing with a minimally disruptive update that preserves a lower bound on the log-determinant of the experts' weighted output span. The router still tracks the desired mixture, but a projection prevents the active experts from becoming linearly redundant or collapsing onto a low-rank subset.

## Mathematical statement

The paper defines the authority co-metric at a sweet spot as $S:=D(v^{\star})/4=\sum_i\psi_i^{\star}A_{\bullet i}A_{\bullet i}^{\top}$, where $A_{\bullet i}\in\mathbb{R}^{m}$ is actuator column $i$, $\psi_i^{\star}\geq0$ is its capacity weight, and $m$ is the wrench dimension. Adapt this to a router with expert direction vectors $a_i\in\mathbb{R}^{d}$ and nonnegative gate weights $\psi_i(p)$ by defining $S(p)=\sum_{i=1}^{n}\psi_i(p)a_i a_i^{\top}+\epsilon I$ and authority $L(p)=\log\det S(p)$. The safe set is $\mathcal{C}=\{p:L(p)\geq\ell^{\dagger}\}$, where $\ell^{\dagger}$ is a chosen authority floor. The dropout theorem gives the exact gap $\Delta_k=-\ln(1-\lambda_k)$, where $\lambda_k=\psi_k a_k^{\top}S^{-1}a_k$ is the weighted leverage of expert $k$; large leverage means that masking this expert causes a large authority loss. For symmetric components, $\lambda_k=m/n$ and $\Delta=\ln(n/(n-m))$. The paper's viability inequality, $\sup_{|\tau|\leq\bar{\tau}}\nabla h(v)^{\top}(d(v)+M^{-1}\tau)\geq\sum_i|\partial_i h|\bar a_i\geq0$, motivates enforcing nonnegative first-order barrier growth under bounded router-logit updates. Use the discrete barrier condition $L(p_{t+1})\geq L(p_t)-\eta_b(L(p_t)-\ell^{\dagger})$, with $0<\eta_b\leq1$.

## Key formulas

- $$S:=D(v^{\star})/4=\sum_{i=1}^{n}\psi_i^{\star}A_{\bullet i}A_{\bullet i}^{\top},\qquad L=\log\det S.$$
- $$\Delta_k=-\ln(1-\lambda_k),\qquad \lambda_k=\psi_k^{\star}A_{\bullet k}^{\top}S^{-1}A_{\bullet k}\in(0,1],\qquad \sum_{k=1}^{n}\lambda_k=m.$$
- $$\Delta=\ln\!\left(\frac{n}{n-m}\right)\quad\text{for symmetric designs with }\lambda_k=m/n.$$
- $$\min_{\delta z}\;\frac12\|\delta z-u\|_2^2\quad\text{s.t.}\quad \nabla_zL(z)^{\top}\delta z\geq-\eta_b\bigl(L(z)-\ell^{\dagger}\bigr),\quad |\delta z_j|\leq r_j.$$

## Implementation notes

Integrate the method at the MoE router immediately after computing the nominal router-logit update and before applying it. For each batch, collect expert contribution directions $a_i$: use the mean expert output on the current token batch, center it, and project it to a dimension $d\leq64$; a stronger variant can use Jacobian-vector products of each expert output with respect to its input. Let $p_i$ be the batch-average routing probability and define $\psi_i=p_i(1-p_i)+\epsilon$ or $\psi_i=p_i^2+\epsilon$, with $\epsilon=10^{-4}$. Compute $S=\sum_i\psi_i a_i a_i^T+\epsilon I$, $L=\log\det S$, and $g_L=\nabla_zL$ through autodiff. Given nominal update $u$, authority floor $\ell^\dagger$, barrier rate $\eta_b$, and per-logit bounds $r_j$, solve $\min_\delta\frac12\|\delta-u\|^2$ subject to $g_L^T\delta\geq-\eta_b(L-\ell^\dagger)$ and $-r_j\leq\delta_j\leq r_j$. With only the linear constraint, use the exact projection $\delta=u+\max(0,(b-g_L^Tu)/\|g_L\|^2)g_L$, where $b=-\eta_b(L-\ell^\dagger)$; use a small QP solver when box bounds are active. Set $z\leftarrow z+\delta$ and then compute the normal softmax or top-$k$ routing. Compute from the paper's mathematics the log-det authority, inverse-covariance leverage scores $\lambda_i=\psi_i a_i^TS^{-1}a_i$, dropout gaps $-\log(1-\lambda_i)$, and barrier projection. Estimate empirically the representation dimension, smoothing over batches, gate-weight function, and floor; choose $\ell^\dagger$ as the 10th percentile of authority under a pretrained balanced router. First test a 12-expert Switch-style Transformer on WikiText-103 or a small C4 subset, comparing standard load-balancing loss, entropy regularization, and this projection at equal FLOPs. Track perplexity, authority-floor violations, expert utilization entropy, minimum eigenvalue of $S$, and perplexity after masking the highest-leverage expert. The method succeeds if it reduces collapse and masked-expert degradation with less than 1% routing overhead while matching or improving baseline perplexity.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.