# Fisher–Rao geodesic router

- ID: 2366
- Canonical URL: https://synthcore.org/idea/2366/fisher-rao-geodesic-router
- API JSON: https://synthcore.org/api/idea/2366.json
- API Markdown: https://synthcore.org/api/idea/2366.md
- Verification status: unverified
- Source: [arXiv:2608.15805](https://arxiv.org/abs/2608.15805)
- Category: geometry
- Solves: stability, accuracy, generalization
- ML areas: moe-routing, attention, optimizer
- Math tags: geometry, information-theory, metric-geometry
- Ratings: usefulness 6/10; difficulty 4/10; novelty 4/10

## Idea description

Replace Euclidean updates and interpolation of probability vectors in a mixture-of-experts router or attention simplex with updates in square-root coordinates, where the Fisher–Rao geometry is spherical. If the task has a desired neutral or calibrated family of distributions, represent that family as a linear subsphere in square-root space and project router outputs onto it after every update.

## Mathematical statement

For a probability vector \(\pi=(\pi_1,\ldots,\pi_m)\) in the interior simplex \(\Delta_{m-1}^{\circ}=\{\pi_i>0,\sum_i\pi_i=1\}\), the Fisher–Rao line element is \(ds^2=\sum_i d\pi_i^2/\pi_i\). The square-root embedding \(\Phi(\pi)=2(\sqrt{\pi_1},\ldots,\sqrt{\pi_m})\) maps the simplex to the positive orthant of the Euclidean sphere \(\|\Phi(\pi)\|_2=2\). For two distributions \(\pi,\rho\), the Fisher–Rao distance is twice the spherical angle. A totally geodesic lower-dimensional family is obtained by intersecting this sphere with a fixed linear subspace \(S\subseteq\mathbb R^m\). The projector \(P_S\) is the Euclidean orthogonal projector onto \(S\), and renormalization returns the result to the radius-two sphere.

## Key formulas

- $$ds^2_{FR}=\sum_{i=1}^{m}\frac{d\pi_i^2}{\pi_i},\qquad \Phi(\pi)=2(\sqrt{\pi_1},\ldots,\sqrt{\pi_m}),\qquad \|\Phi(\pi)\|_2=2.$$
- $$d_{FR}(\pi,\rho)=2\arccos\!\left(\sum_{i=1}^{m}\sqrt{\pi_i\rho_i}\right).$$
- $$\operatorname{slerp}(x,y;t)=\frac{\sin((1-t)\theta)}{\sin\theta}x+\frac{\sin(t\theta)}{\sin\theta}y,\qquad \theta=\arccos\frac{x^{\mathsf T}y}{4},\qquad \pi_i(t)=\left(\frac{x_i(t)}{2}\right)^2.$$
- $$x_{\mathrm{proj}}=2\,\frac{P_Sx}{\|P_Sx\|_2},\qquad \pi_{\mathrm{proj},i}=\left(\frac{x_{\mathrm{proj},i}}{2}\right)^2.$$

## Implementation notes

Integrate the method into the router probability tensor immediately after its softmax and before top-k selection or mixture aggregation. Given router logits \(a\in\mathbb R^m\), compute \(\pi=\operatorname{softmax}(a)\), then \(x=2\sqrt{\pi+\varepsilon}\) with \(\varepsilon=10^{-8}\). For an unconstrained Fisher router, renormalize \(x\) to norm 2 and recover probabilities with \(\pi_i=(x_i/2)^2\). For a constrained experiment, choose an orthonormal matrix \(U\in\mathbb R^{m\times r}\), define \(P_S=UU^{\mathsf T}\), compute \(v=P_Sx\), add a small positive floor if some coordinates are negative, renormalize \(x'=2v/\|v\|_2\), and set \(\pi'_i=(x'_i/2)^2\). A differentiable positivity-preserving alternative is \(x=2\sqrt{\operatorname{softmax}(Uh)}\) for a learned low-dimensional vector \(h\). Keep the standard load-balancing loss unchanged. Compare ordinary softmax routing, Euclidean probability projection, and Fisher projection in a 4- or 8-expert Switch-style MLP on CIFAR-10 or a small language-model dataset. Measure validation loss, expert-load variance, router entropy, gradient spikes, and dead-expert frequency. Success is lower load variance or fewer routing instabilities at equal loss, or better validation loss under large router learning rates. Also test spherical interpolation between learned routing distributions and verify constant Fisher speed numerically.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.