Pressure-Based Expert Selection

Add a pressure-shaped auxiliary objective at the router logits of a mixture-of-experts layer. For each minibatch input $x$, run either all experts or a cheap periodically refreshed subset to obtain detached reward estimates $r_i(x)=-\ell_i(x)$, where $\ell_i$ is the expert's per-example task loss. Estimate $q_i(x)$ with one-step Jacobian-vector products or use a running exponential average of positive log singular-value estimates. Set router logits to $(r_i+\gamma q_i)/\alpha$, or train ordinary router logits with a KL penalty toward this target distribution: $L_{route}=\operatorname{KL}(p_\theta(\cdot\mid x)\|p^*(\cdot\mid x))$. Use stop-gradient on rewards and expansion estimates to avoid second-order instability. Pseudocode: evaluate expert losses, compute $u_i=r_i+\gamma q_i$, calculate $p_i^*=\operatorname{softmax}(u/\alpha)$, calculate router probabilities $p_\theta$, add $\lambda L_{route}$ to task loss, and update parameters. The pressure identity is taken from the paper's variational-principle machinery; the Gibbs router and neural reward estimates are the constructive adaptation. First cheap experiment: use four experts on CIFAR-10 or a synthetic two-cluster classification task, compare standard softmax routing against pressure routing while sweeping $\alpha$. The predicted bifurcation is a sharp concentration transition when the reward gap $\Delta r$ exceeds approximately $\alpha\log(K-1)$: the best-expert probability should exceed $0.9$ near that boundary, while below it route entropy should remain substantial. Measure whether this transition and the entropy-versus-reward slope match the Gibbs prediction.