# Curvature-Calibrated Exponential Expert Averaging

- ID: 358
- Canonical URL: https://synthcore.org/idea/358/curvature-calibrated-exponential-expert-averaging
- API JSON: https://synthcore.org/api/idea/358.json
- API Markdown: https://synthcore.org/api/idea/358.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.02247](https://arxiv.org/abs/2607.02247)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: moe-routing, fine-tuning, regularization
- Math tags: convex-analysis, statistics, optimization, probability
- Ratings: usefulness 6/10; difficulty 4/10; novelty 3/10

## Idea description

Replace an unconstrained softmax gate over a finite set of neural experts with exponential weights whose temperature is chosen to satisfy the paper's explicit stability condition. The goal is to prevent low-temperature expert collapse while retaining the model-selection rate when the expert losses are bounded and strongly convex in the prediction.

## Mathematical statement

For a finite dictionary of predictors f_1,...,f_M, the aggregation estimator is \widehat f_T(x)=\sum_{j=1}^M w_j(T)f_j(x), with weights w_j(T)=\exp(-\widehat R_j/T)/\sum_{k=1}^M\exp(-\widehat R_k/T), where \widehat R_j=(1/n)\sum_{i=1}^n\ell(f_j(X_i),Y_i) is empirical risk and T>0 is temperature. The paper's upper-bound condition is (L^2/T)\exp(B/T)\leq\mu/2, where B bounds the loss 0\leq\ell\leq B, L is the loss Lipschitz constant in the prediction argument, and \mu is the strong-convexity constant of the loss. Under this condition, expected excess risk is bounded at scale T\log(M)/(n+1). For squared loss with predictions and labels in [0,b], the paper states that T\geq4b^2 suffices. Proposition 1 supplies complementary lower-bound behavior: for every T, some problem has excess risk at least \gamma_M\min\{1,T\log(M-1)/n\}, with \gamma_M\in[1/4,1] and \gamma_M    o1. Thus temperatures that are too small can be suboptimal despite producing sharper model selection.

## Key formulas

- $$w_j(T)=\frac{\exp\!\left(-\widehat R_j/T\right)}{\sum_{k=1}^{M}\exp\!\left(-\widehat R_k/T\right)},\qquad \widehat f_T(x)=\sum_{j=1}^{M}w_j(T)f_j(x),\qquad \widehat R_j=\frac{1}{n}\sum_{i=1}^{n}\ell(f_j(X_i),Y_i).$$
- $$\frac{L^{2}}{T}\exp\!\left(\frac{B}{T}\right)\leq\frac{\mu}{2}.$$
- $$\mathbb{E}\left[\mathcal R_P(\widehat f_T)-\min_{1\leq j\leq M}\mathcal R_P(f_j)\right]\ \lesssim\ \frac{T\log M}{n+1}.$$
- $$\mathbb{P}\!\left(\mathcal R_P(\widehat f_T)\geq\min_j\mathcal R_P(f_j)+\gamma_M\min\left\{1,\frac{T\log(M-1)}{n}\right\}\right)=1.$$

## Implementation notes

(1) Integration point: use this as a prediction-level aggregation layer over M frozen or trainable neural experts, such as LoRA adapters, checkpoints, small subnetworks, or MoE experts. On each validation minibatch, compute every expert's prediction and scalar loss; do not apply the weights to hidden states initially. For squared loss with output and target clipping to [0,b], initialize T=4b^2. For general bounded losses, estimate B from the maximum clipped minibatch loss, estimate L by the maximum finite-difference slope of loss versus prediction on a calibration batch, and use a conservative curvature estimate mu from the loss definition; increase T until (L^2/T)*exp(B/T) <= mu/2. (2) Pseudocode: maintain cumulative losses c[j]=0. For each calibration batch (x,y), compute pred[j]=expert_j(x), batch_loss[j]=mean(loss(pred[j],y)), update c[j] <- c[j]+batch_loss[j], set logits[j] <- -c[j]/T, weights <- softmax(logits), and output pred <- sum_j weights[j]*pred[j]. Recompute weights every K steps or once per epoch. Apply a floor w[j] >= epsilon followed by renormalization if experts are jointly trained. (3) The inequality, the T=4b^2 rule, and the predicted risk scale T log(M)/(n+1) come from the paper; B, L, mu, and the effective sample count n are estimated or known from the implementation. (4) First experiment: CIFAR-10 with M=8 ResNet-18 checkpoints or adapters, compare uniform averaging, ordinary tuned softmax temperatures, T=0.1, and the calibrated temperature. Track validation accuracy, expert entropy, worst-seed variance, and cumulative excess loss. Success is calibrated aggregation avoiding collapse and matching or improving the best expert, especially under distribution shift, with no more than a small inference overhead.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a prediction-level cumulative exponential expert aggregator with stable softmax, optional weight floors, curvature-condition checks, and a 20-seed frozen-expert regression experiment. T=4b² satisfies the condition for half-squared loss (0.283 <= 0.5) but not the literal full squared-loss normalization (1.284 > 1), whose minimum temperature is approximately 4.905b². In the experiment, calibrated T=2.452 achieved loss 0.003374 versus uniform 0.003378, while low-temperature T=0.01 was substantially better at 0.002978, so the proposed calibration did not produce the promised accuracy win.

### Mechanism check

- Verdict: Built a prediction-level cumulative exponential expert aggregator with stable softmax, optional weight floors, curvature-condition checks, and a 20-seed frozen-expert regression experiment. T=4b² satisfies the condition for half-squared loss (0.283 <= 0.5) but not the literal full squared-loss normalization (1.284 > 1), whose minimum temperature is approximately 4.905b². In the experiment, calibrated T=2.452 achieved loss 0.003374 versus uniform 0.003378, while low-temperature T=0.01 was substantially better at 0.002978, so the proposed calibration did not produce the promised accuracy win.
- Confidence: 9/10
- Limitations: Only a small synthetic bounded regression task with eight frozen stochastic experts was tested; no neural networks, CIFAR-10, trainable experts, distribution shift, FLOP or latency measurement, or large-scale generalization study was performed.

## Artifacts

- [experiment.py](https://synthcore.org/code/85/experiment.py)
- [report.md](https://synthcore.org/code/85/report.md)
- [Download all files as ZIP](https://synthcore.org/download/85)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.