# Gaussian Extreme-Logit Calibration

- ID: 1236
- Canonical URL: https://synthcore.org/idea/1236/gaussian-extreme-logit-calibration
- API JSON: https://synthcore.org/api/idea/1236.json
- API Markdown: https://synthcore.org/api/idea/1236.md
- Verification status: unverified
- Source: [arXiv:2607.21392](https://arxiv.org/abs/2607.21392)
- Category: stability
- Solves: stability, accuracy
- ML areas: attention, moe-routing, transformer
- Math tags: probability, convex-analysis, statistics
- Ratings: usefulness 5/10; difficulty 4/10; novelty 6/10

## Idea description

Normalize attention or router logits and control their upper tail using the paper's sharper Gaussian-maximum exponent rather than a correlation-blind sub-Gaussian bound. Use the resulting threshold to add a soft penalty or adaptive temperature whenever the observed maximum exceeds the calibrated level, reducing rare one-token or one-expert domination.

## Mathematical statement

For a centered Gaussian vector X=(X_1,...,X_N) with max_i Var(X_i) <= 1, the Borell-Sudakov-Tsirelson inequality gives P(max_i X_i >= E max_i X_i + t) <= exp(-t^2/2). The paper's sharper result assumes fixed alpha in (0,sqrt(2)) and kappa>0 such that E max_i X_i >= alpha sqrt(log N) and E max_i X_i + kappa sqrt(log N) <= sqrt(2 log N), and proves P(max_i X_i >= E max_i X_i + kappa sqrt(log N)) <= N^(-kappa^2/(2-alpha^2)+o(1)). Here N is the number of competing logits, mu=E[max_i X_i] is their expected maximum, alpha=mu/sqrt(log N) is the normalized expected maximum, and kappa=(u-mu)/sqrt(log N) for threshold u. The transferable property is that the tail exponent depends on the observed baseline maximum through alpha, thereby incorporating correlation structure indirectly. For neural logits z, center and divide by a scale s so x_i=(z_i-mean(z))/s has approximately unit marginal variance, then use the theorem as a calibration heuristic.

## Key formulas

- $$\mathbb{P}\bigl(\max_{1\leq i\leq N}X_i\geq\mathbb{E}\max_{1\leq i\leq N}X_i+t\bigr)\leq e^{-t^{2}/2},\qquad t\geq 0.$$
- $$\mathbb{P}\left(\max_i X_i\geq\mathbb{E}\max_i X_i+\kappa\sqrt{\log N}\right)\leq N^{-\kappa^{2}/(2-\alpha^{2})+o(1)},$$ $$\mathbb{E}\max_iX_i\geq\alpha\sqrt{\log N},\qquad \mathbb{E}\max_iX_i+\kappa\sqrt{\log N}\leq\sqrt{2\log N}.$$
- $$\widehat{p}_{\mathrm{ext}}(u)=N^{-\widehat{\kappa}^{\,2}/(2-\widehat{\alpha}^{\,2})},\qquad \widehat{\alpha}=\frac{\widehat{\mu}}{\sqrt{\log N}},\qquad \widehat{\kappa}=\frac{u-\widehat{\mu}}{\sqrt{\log N}},\qquad \widehat{\mu}=\frac{1}{B}\sum_{b=1}^{B}\max_i x_{b,i}.$$
- $$u_{\mathrm{target}}=\widehat{\mu}+\sqrt{(2-\widehat{\alpha}^{\,2})\,\log N\,\log(1/\delta)}.$$

## Implementation notes

Integrate the method at the pre-softmax logit tensor of a transformer attention head or MoE router. For each batch element, let z[b,i] be the N competing logits. Compute m[b]=mean_i z[b,i] and s[b]=sqrt(mean_i(z[b,i]-m[b])^2+epsilon), then set x[b,i]=(z[b,i]-m[b])/stop_gradient(s[b]). Maintain an exponential moving average mu_hat of mean_b max_i x[b,i]. Compute alpha_hat=clip(mu_hat/sqrt(log(N)),0,sqrt(2)-margin). For a target extreme-event probability delta, calculate u_target=mu_hat+sqrt((2-alpha_hat^2)*log(N)*log(1/delta)). Add penalty=lambda*mean_b softplus(max_i x[b,i]-u_target)^2 to the task loss, or use an adaptive temperature T[b]=max(1,max_i x[b,i]/u_target) and replace z[b,i] by z[b,i]/T[b]. Pseudocode: x=normalize(z); mu=EMA(mean(max(x,dim=-1))); alpha=clip(mu/sqrt(logN),0,sqrt(2)-0.05); u=mu+sqrt((2-alpha^2)*logN*log(1/delta)); penalty=mean(softplus(max(x)-u)^2); loss=task_loss+lambda*penalty. The theorem supplies the tail exponent and threshold form; mu, marginal scale, and violation rate are estimated empirically. Start with a small Switch-Transformer router and a four-head attention model on WikiText-2 or CIFAR-10, comparing standard RMSNorm and fixed temperature against this method at equal FLOPs. Record maximum-logit violation frequency, attention entropy, MoE load imbalance, gradient-norm spikes, validation loss, and accuracy. Success means fewer extreme winners and fewer optimization spikes without destroying useful sparse routing, with lower validation loss or better accuracy at the same training budget.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.