# Spectator-canceling curvature router

- ID: 1618
- Canonical URL: https://synthcore.org/idea/1618/spectator-canceling-curvature-router
- API JSON: https://synthcore.org/api/idea/1618.json
- API Markdown: https://synthcore.org/api/idea/1618.md
- Verification status: unverified
- Source: [arXiv:2608.00041](https://arxiv.org/abs/2608.00041)
- Category: architecture
- Solves: accuracy, stability, generalization
- ML areas: moe-routing, transformer, training-dynamics
- Math tags: dynamical-systems, differential-geometry, linear-algebra, probability
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Replace or augment a mixture-of-experts router with a relative transverse-curvature score computed between experts, rather than relying only on the router MLP logits. Experts that provide a broader, less stiff local response in task-relevant directions receive higher routing probability, while common nuisance or spectator directions cancel from the comparison. The score is invariant under a common linear reparameterization of the routing coordinates and can be restricted to a low-dimensional subspace so it is practical.

## Mathematical statement

The paper writes a Markov path weight as $P[\gamma]\propto\exp(\tfrac12\sigma[\gamma]-K[\gamma])$, where $\gamma$ is a path, $\sigma$ is the time-antisymmetric entropy flux, and $K$ is the time-symmetric dynamical activity or frenesy. In the constructed gradient-flow case, the entropy difference between channels is zero, so selection is controlled by the geometric frenesy term. For two channels with transverse Hessians $H_{\perp,k}=H^{\rm spec}\oplus H^{\rm react}_k$, the discriminating density is $\Delta=\tfrac12\ln\frac{\det(D_\perp H_{\perp,2})}{\det(D_\perp H_{\perp,1})}=\tfrac12\ln\det(H_{\perp,1}^{-1}H_{\perp,2})$. Here $H_{\perp,k}$ is the positive-definite curvature matrix of branch $k$ in directions orthogonal to the shared route, $D_\perp$ is a positive-definite diffusion or metric matrix, and $H^{\rm spec}$ is a block shared by both branches. The shared spectator determinant cancels exactly; if $D_\perp$ is common, it cancels too. Equivalently, $\Delta=\tfrac12\sum_i\ln\lambda_i$, where $\lambda_i$ are generalized eigenvalues satisfying $H_{\perp,2}v_i=\lambda_iH_{\perp,1}v_i$. Adapt this as a router score $r_k(x)=-\tfrac12\log\det(\widetilde H_k(x))$, or relative to a reference expert $j$, $r_k-r_j=\tfrac12\log\det(\widetilde H_j^{-1}\widetilde H_k)$, with $\widetilde H_k$ a damped positive curvature estimate. The entropy/frenesy split is not itself used as a loss; it motivates testing whether curvature-based branch selection adds information when ordinary value-based logits are held fixed.

## Key formulas

- $$P[\gamma]\;\propto\;\exp\!\bigl(\tfrac{1}{2}\sigma[\gamma]-K[\gamma]\bigr)$$
- $$\Delta(s)\equiv\tfrac{1}{2}\ln\frac{\det\bigl(D_{\perp}H_{\perp,2}\bigr)}{\det\bigl(D_{\perp}H_{\perp,1}\bigr)}=\tfrac{1}{2}\ln\det\bigl(H_{\perp,1}^{-1}H_{\perp,2}\bigr)$$
- $$H_{\perp,k}=H^{\rm spec}\oplus H^{\rm react}_{k},\qquad \Delta=\tfrac12\sum_i\ln\lambda_i,\quad H_{\perp,2}v_i=\lambda_iH_{\perp,1}v_i$$
- $$p(k\mid x)=\operatorname{softmax}_k\!\left(a_k(x)+\alpha r_k(x)\right),\qquad r_k(x)=-\tfrac12\log\det\!\left(U^\top\widetilde H_k(x)U\right)$$

## Implementation notes

Integrate this into the router of a top-1 or top-2 MoE Transformer, immediately before the softmax that converts router logits into expert probabilities. Let $z(x)\in\mathbb R^d$ be the router hidden state and let $U\in\mathbb R^{d\times q}$ be a fixed or slowly updated orthonormal matrix selecting a small routing subspace, with $q=8$ or $16$. For each expert $k$, define a scalar proxy loss $\ell_k(z)$ on the current minibatch: use the token-level cross-entropy after expert $k$, or use the squared prediction error between the expert output and a detached teacher or ensemble output. Estimate curvature in the subspace by a Gauss-Newton or empirical Fisher matrix, $\widetilde H_k=U^\top(J_k^\top J_k/B+\rho I)U$, where $J_k$ is the Jacobian of the scalar residuals with respect to $z$, $B$ is the number of sampled tokens, and $\rho>0$ is damping. Compute $\log\det\widetilde H_k$ using Cholesky, set $r_k=-\frac12\log\det\widetilde H_k$, and add $\alpha r_k$ to the existing router logit $a_k$. Use stop-gradient through the curvature estimate for the first MVP to avoid second-order training costs; refresh estimates every 16-64 optimizer steps and maintain an exponential moving average. Pseudocode: sample tokens; compute each selected expert output; form residuals; obtain Jacobian-vector products or a low-rank Gauss-Newton estimate; project with $U$; add $\rho I$; compute the Cholesky log-determinant; update $p=\operatorname{softmax}(a+\alpha r)$; dispatch top-2 experts; update the EMA. The paper's exact cancellation applies when $H_k$ has a common spectator block; empirically verify this by appending identical random nuisance coordinates to $z$ and checking that relative scores remain nearly unchanged. The first experiment should use a 125M-parameter Transformer on WikiText-103 or a synthetic two-task classification dataset with 8 experts, comparing standard load-balanced routing, curvature-only routing, and additive routing at matched FLOPs. Measure validation loss, expert-load coefficient of variation, routing entropy, and extra router latency. Success means lower loss or better calibration at equal compute, stable routing under nuisance-coordinate reparameterization, and little degradation when $q$ is increased; a stronger signal is an improvement that persists after adding shared stiff directions to every expert.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.