Gaussian Extreme-Logit Calibration
Implementation & benchmark of arXiv:2607.21392 — Gaussian Convexity Principles for Sharp Moderate Deviations of Gaussian Maxima and Critical SK Free Energy Variance
Source paper: Gaussian Convexity Principles for Sharp Moderate Deviations of Gaussian Maxima and Critical SK Free Energy Variance arXiv:2607.21392 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Normalize attention or router logits and control their upper tail using the paper's sharper Gaussian-maximum exponent rather than a correlation-blind sub-Gaussian bound. Use the resulting threshold to add a soft penalty or adaptive temperature whenever the observed maximum exceeds the calibrated level, reducing rare one-token or one-expert domination.
Formulas
Mathematical statement
For a centered Gaussian vector X=(X_1,...,X_N) with max_i Var(X_i) <= 1, the Borell-Sudakov-Tsirelson inequality gives P(max_i X_i >= E max_i X_i + t) <= exp(-t^2/2). The paper's sharper result assumes fixed alpha in (0,sqrt(2)) and kappa>0 such that E max_i X_i >= alpha sqrt(log N) and E max_i X_i + kappa sqrt(log N) <= sqrt(2 log N), and proves P(max_i X_i >= E max_i X_i + kappa sqrt(log N)) <= N^(-kappa^2/(2-alpha^2)+o(1)). Here N is the number of competing logits, mu=E[max_i X_i] is their expected maximum, alpha=mu/sqrt(log N) is the normalized expected maximum, and kappa=(u-mu)/sqrt(log N) for threshold u. The transferable property is that the tail exponent depends on the observed baseline maximum through alpha, thereby incorporating correlation structure indirectly. For neural logits z, center and divide by a scale s so x_i=(z_i-mean(z))/s has approximately unit marginal variance, then use the theorem as a calibration heuristic.
Implementation notes
Integrate the method at the pre-softmax logit tensor of a transformer attention head or MoE router. For each batch element, let z[b,i] be the N competing logits. Compute m[b]=mean_i z[b,i] and s[b]=sqrt(mean_i(z[b,i]-m[b])^2+epsilon), then set x[b,i]=(z[b,i]-m[b])/stop_gradient(s[b]). Maintain an exponential moving average mu_hat of mean_b max_i x[b,i]. Compute alpha_hat=clip(mu_hat/sqrt(log(N)),0,sqrt(2)-margin). For a target extreme-event probability delta, calculate u_target=mu_hat+sqrt((2-alpha_hat^2)*log(N)*log(1/delta)). Add penalty=lambda*mean_b softplus(max_i x[b,i]-u_target)^2 to the task loss, or use an adaptive temperature T[b]=max(1,max_i x[b,i]/u_target) and replace z[b,i] by z[b,i]/T[b]. Pseudocode: x=normalize(z); mu=EMA(mean(max(x,dim=-1))); alpha=clip(mu/sqrt(logN),0,sqrt(2)-0.05); u=mu+sqrt((2-alpha^2)*logN*log(1/delta)); penalty=mean(softplus(max(x)-u)^2); loss=task_loss+lambda*penalty. The theorem supplies the tail exponent and threshold form; mu, marginal scale, and violation rate are estimated empirically. Start with a small Switch-Transformer router and a four-head attention model on WikiText-2 or CIFAR-10, comparing standard RMSNorm and fixed temperature against this method at equal FLOPs. Record maximum-logit violation frequency, attention entropy, MoE load imbalance, gradient-norm spikes, validation loss, and accuracy. Success means fewer extreme winners and fewer optimization spikes without destroying useful sparse routing, with lower validation loss or better accuracy at the same training budget.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.