Effective-resistance natural-gradient routing
Implementation & benchmark of arXiv:2607.13990 — Effective Resistance in Fixed-Rank External-Field Measures and Constant-Stretch Correlated Sampling on the Hypersimplex
Source paper: Effective Resistance in Fixed-Rank External-Field Measures and Constant-Stretch Correlated Sampling on the Hypersimplex arXiv:2607.13990 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace independent expert activation or ordinary softmax routing with an exact fixed-m external-field subset router. Parameterize expert weights by logits, use the subset covariance as the Fisher matrix, and precondition router gradients with its Moore-Penrose pseudoinverse on the sum-zero subspace. The paper's resistance bound supplies a data-dependent ceiling for pairwise logit updates, preventing unstable motion when some experts have low inclusion variance.
Formulas
Mathematical statement
For d experts and a fixed routing budget m, the external-field distribution is p_theta(S)=exp(sum_{i in S} theta_i)/e_m(exp(theta)) over subsets S of [d] with |S|=m, where theta_i are expert logits, w_i=exp(theta_i)>0, and e_m(w)=sum_{|T|=m} product_{l in T}w_l. Let X=I_S in {0,1}^d, mu=E[X], and Sigma=Cov(X). Because 1_d^T X=m always, Sigma 1_d=0 and only logit differences are identifiable. The main theorem states that for i not equal to j, (e_i-e_j)^T Sigma^dagger(e_i-e_j) <= 1/v_i+1/v_j, where Sigma^dagger is the Moore-Penrose pseudoinverse, e_i is the i-th standard basis vector, and v_i=Sigma_ii=mu_i(1-mu_i). Its corollary is Sigma >= 1/2(D-vv^T/V), with D=diag(v), v=(v_1,...,v_d)^T, and V=sum_i v_i. For router loss J(theta), g=nabla_theta J, use the constrained natural-gradient direction u=Sigma^dagger P g, where P=I-1_d 1_d^T/d projects out the unidentifiable all-ones direction. Scale the update so every pairwise logit displacement obeys |Delta theta_i-Delta theta_j| <= rho sqrt(1/v_i+1/v_j).
Implementation notes
Integrate this into the router-logit update, not into the transformer hidden-state path. For each token or token group, let the router produce logits theta in R^d, choose a fixed budget m, and sample S from the external-field law. A practical exact sampler uses dynamic programming for elementary symmetric polynomials: compute E[k][r] for the first k experts with E[0][0]=1 and E[k][r]=E[k-1][r]+exp(theta_k)E[k-1][r-1], then sample the subset backward conditional on the remaining cardinality. For small d, enumerate all m-subsets; for larger d, use the DP conditional probabilities. At each optimizer step, compute inclusion means mu_i=exp(theta_i)e_{m-1}(w_{-i})/e_m(w), variances v_i=mu_i(1-mu_i), and pairwise probabilities P_ij=exp(theta_i+theta_j)e_{m-2}(w_{-i,-j})/e_m(w), giving Sigma_ij=P_ij-mu_i mu_j for i not equal to j. Project the router gradient g with P, solve Sigma u=P g using conjugate gradient restricted to the sum-zero subspace, and use u as the natural-gradient direction. Add the resistance-based trust-region rescaling in the final formula; rho is a tunable maximum pairwise logit movement. Estimate Sigma from exact DP marginals when d is modest, or from a moving-average minibatch covariance for large or hierarchical routers. First test on a 12-expert, top-2 or top-4 CIFAR-100 or WikiText MoE, comparing ordinary softmax/top-m routing with Adam at equal router FLOPs. Measure validation loss, expert-load coefficient of variation, dead-expert fraction, gradient-norm spikes, and loss variance. The expected signal is fewer routing-collapse events and faster loss reduction at equal communication budget, especially when logits become highly imbalanced.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Confirmed ✓
Stage 2 · Practical benchmark result: Significantly worse ✗
Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built an exact fixed-m external-field router with DP subset sampling, exact covariance/Fisher computation, pseudoinverse natural-gradient routing, and resistance-based pairwise trust scaling. The mechanism manifested: across 65 imbalanced configurations, the maximum normalized resistance ratio was 0.999996, the covariance lower-bound violation was only numerical noise (minimum eigenvalue -2.0e-14), and trust-cap sweeps matched predicted min(rho, raw_step) to machine precision. In the toy optimization, natural-trust reached loss 1.03e-7 versus 1.47e-3 for vanilla after 80 steps, though this is not evidence of a production MoE win.
- Agent confidence
- 9/10
- Baseline
- Toy exact-router optimization: final loss 0.001474, step-10 loss 0.05923, step-40 loss 0.005067, final load CV 0.6995
- Idea
- Toy exact-router natural-gradient plus rho=0.35 cap: final loss 1.034e-7, step-10 loss 0.001425, step-40 loss 8.850e-6, final load CV 0.7296; resistance ratio max 0.999996; covariance-bound minimum eigenvalue -2.0e-14; trust-cap observed/predicted equal for rho={0.01,0.03,0.1,0.3,1.0}; DP sampling max marginal error 0.00119 over 200000 samples
Limitations: No CIFAR, language modeling, transformer, expert compute, communication-cost, or minibatch-load experiment was tested. Exact covariance uses subset enumeration and is practical only for small expert counts; the DP sampler is implemented, but DP marginal computation for large d and a CUDA training integration were not implemented.
How to run: python3 experiment.py > results.json && /home/maxwelhelp/main/bin/python3 verify_router.py
Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.13990)
Significantly WORSE than the tuned baseline (+10.1%, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Router_regime_regression
- Model
- custom_mlp_moe
- Paired seeds
- 8
- Baseline mean
- 0.3103
- Idea mean
- 0.3417
- Effect (Δ)
- 0.0315 (+10.1%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±5.3%
- Mechanism
- Confirmed ✓
- Practical effect
- Hurts
- Baseline tuning
- swept over 6 configs
Only the registered custom 400-sample router-regression track was tested. Built-in CIFAR, language-modeling, sequence, and dynamics tracks were not structurally matched to expert routing and were not used. Large-scale MoE, communication cost, expert-load CV, dead-expert fraction, and gradient-spike analysis were not tested. The benchmark used six small experts and exact subset enumeration, so scalability was not evaluated.
How to run: python3 run_bench.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 9/10
The registered custom router_regime_regression benchmark compared matched end-to-end six-expert MoE systems: standard softmax routing with Adam against fixed-m external-field routing with exact covariance pseudoinverse router gradients and resistance trust scaling. The trained-model mechanism signature was confirmed and the math sanity checks passed. Nevertheless, the best idea result was significantly worse than the tuned baseline on test MSE, so this idea does not provide a benchmark win.
- Baseline
- Mean test MSE 0.3102635518 over 8 seeds, std 0.0749747019; best lr=0.012, temp=0.7.
- Idea
- Best mean test MSE 0.3417467810 over 8 seeds, std 0.0732672876; lr=0.012, temp=0.7, rho=0.3. Paired delta +0.0314832292, p=0.0081, 0/8 wins.
- Math sanity
- max resistance ratio=0.9998409991; minimum covariance-bound eigenvalue=-1.186e-16; trust-cap maximum absolute error=2.776e-17.
- Mechanism signature
- confirmed=true; mean observed cap=0.2041564326 and mean predicted cap=0.2041564326.
Benchmark result
Significantly WORSE than the tuned baseline (+10.1%, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Router_regime_regression
- Model
- custom_mlp_moe
- Paired seeds
- 8
- Baseline mean
- 0.3103
- Idea mean
- 0.3417
- Effect (Δ)
- 0.0315 (+10.1%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±5.3%
- Mechanism
- Confirmed ✓
- Practical effect
- Hurts
- Baseline tuning
- swept over 6 configs
Only the registered custom 400-sample router-regression track was tested. Built-in CIFAR, language-modeling, sequence, and dynamics tracks were not structurally matched to expert routing and were not used. Large-scale MoE, communication cost, expert-load CV, dead-expert fraction, and gradient-spike analysis were not tested. The benchmark used six small experts and exact subset enumeration, so scalability was not evaluated.
How to run: python3 run_bench.py
Artifacts
- bench_report.json 5.7 KB View Raw
- custom_router_track.py 1.0 KB View Raw
- experiment.py 3.7 KB View Raw
- report.md 1.8 KB View
- report_bench_2026-08-31T100902.md 5.9 KB View
- results.json 2.9 KB View Raw
- router.py 2.5 KB View Raw
- run_bench.py 7.3 KB View Raw
- verify_router.py 0.8 KB View Raw