# Fisher-Geometry Expert Menu

- ID: 84
- Canonical URL: https://synthcore.org/idea/84/fisher-geometry-expert-menu
- API JSON: https://synthcore.org/api/idea/84.json
- API Markdown: https://synthcore.org/api/idea/84.md
- Verification status: unverified
- Source: [arXiv:2608.22905](https://arxiv.org/abs/2608.22905)
- Category: architecture
- Solves: memory, scalability, accuracy
- ML areas: moe-routing, quantization, fine-tuning, inference-speedup
- Math tags: statistics, information-theory, metric-geometry, approximation-theory
- Ratings: usefulness 5/10; difficulty 6/10; novelty 4/10

## Idea description

Construct a finite menu of experts or LoRA adapters by quantizing the oracle action manifold under a task-aware Fisher-information distortion rather than Euclidean parameter distance. The router can choose an installed expert after observing the input, but only k experts are stored or evaluated. The paper's frontier rate gives a falsifiable accuracy-versus-menu-size prediction.

## Mathematical statement

The paper defines D(q,a) = b(q)^2 times the difference between I(q,a)^(-1) and I(q,a_star(q))^(-1), where q is a context, a is an action or experiment, a_star(q) is the oracle action, I(q,a) is one-observation Fisher information, and b(q) is the derivative of the context-dependent target. The k-menu frontier is A_k = inf over menus V with at most k actions of the supremum over q of the minimum over a in V of D(q,a). Under uniformly quadratic distortion on an Ahlfors-regular oracle image of intrinsic dimension r, the paper gives A_k proportional to k^(-2/r), and the corresponding original-scale excess mean squared error is proportional to n^(-1) k^(-2/r). For neural networks, let a index an expert and let p_a(y given q) be its predictive distribution. Estimate Fisher information by I_hat(q,a) = epsilon + B^(-1) times the sum over calibration examples of the squared norm of the gradient of log p_a(y_j given q_j) with respect to the expert parameters. Define D_hat(q,a) = b_hat(q)^2 times the positive part of I_hat(q,a)^(-1) minus the minimum inverse information over candidate experts.

## Key formulas

- $$D(q,a)=b(q)^{2}\{I(q,a)^{-1}-I(q,a_{\star}(q))^{-1}\}.$$
- $$A_{k}=\inf_{1\leq |V|\leq k}\sup_{q\in\mathcal{Q}}\min_{a\in V}D(q,a).$$
- $$A_k\asymp k^{-2/r},\qquad \operatorname{excess\ MSE}\asymp n^{-1}k^{-2/r}.$$
- $$\widehat I(q,a)=\epsilon+\frac{1}{B}\sum_{j=1}^{B}\left\|\nabla_{\theta_a}\log p_a(y_j\mid q_j)\right\|_2^2,\qquad \widehat D(q,a)=\widehat b(q)^2\left[\widehat I(q,a)^{-1}-\min_{a'}\widehat I(q,a')^{-1}\right]_+.$$

## Implementation notes

Integrate the method at the expert-selection and model-storage layer of a small mixture-of-experts classifier or a collection of LoRA adapters. First train M much larger than k candidate experts on different tasks, data shards, or bootstrap samples, then freeze them for the menu-construction experiment. On a calibration set, compute every candidate's predictive loss and score norm s(j,a), where s(j,a) is the squared norm of the gradient of log p_a(y_j given q_j) with respect to the candidate's parameters. Maintain an exponential moving average and set I_hat(q_j,a) = epsilon plus the moving average of s(j,a). Estimate the oracle candidate by selecting the expert with lowest validation loss for each context. Use b_hat(q_j)^2 equal to the squared gradient norm of the shared-trunk loss with respect to its representation, or use one if this sensitivity estimate is unavailable. Compute D_hat using the displayed formula. Select k experts by farthest-point initialization under D_hat, followed by alternating assignment of each calibration context to its lowest-distortion selected expert and replacement of each selected expert by the candidate that minimizes the maximum assigned distortion. Train a lightweight router to predict these assignments. The mathematical quantities are the distortion and the predicted k^(-2/r) frontier; Fisher estimates, oracle labels, and router training are empirical approximations. Start with CIFAR-100 or a small multi-task image benchmark, M = 64 LoRA adapters, and k in {4, 8, 16, 32}. Compare against random selection, parameter-space k-means, and magnitude-based adapter pruning. Measure validation loss, stored parameters, and inference FLOPs. Success means lower loss at equal k or an approximately linear log-frontier slope near -2/r.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.