Log-Correlated Extreme-Value Logit Regularizer
Implementation & benchmark of arXiv:2607.02233 — Black Holes and Random Variables
Source paper: Black Holes and Random Variables arXiv:2607.02233 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Calibrate the maximum attention logit in each head against the log-correlated extreme-value law instead of applying fixed clipping or a fixed max-norm penalty. Penalize only maxima that exceed the predicted log N minus three-quarter log log N baseline by an unusually large order-one fluctuation, allowing ordinary sharp attention while suppressing rare pathological spikes.
Formulas
Mathematical statement
The paper considers a centered spectral field X_{N,U}(z), where z indexes a domain D containing O(N) microscopic degrees of freedom and U is a statistical universality class. Its conjectured maximum law is sup_{z in D} X_{N,U}(z) = C_U(log N - 3/4 log log N + Y_{N,U}), with C_U independent of N and Y_{N,U} = O_P(1) converging in distribution to Y_{infinity,U}. The limiting fluctuation has upper tail P(Y_{infinity,U} > y) approximately y exp(-2y) as y tends to infinity. Adapt z to query-key index pairs in one attention head, N to the effective number of logits being compared, and X to centered, per-head normalized logits. Use m_N = C(log N - 3/4 log log N) as the expected extreme baseline and penalize the excess Y = (smax(X) - m_N)/C according to the extreme-tail threshold. The paper does not establish that neural logits are log-correlated, so C, the effective N, and the applicability of the law must be estimated empirically.
Implementation notes
Integrate the method after attention-logit construction and before the usual row-wise softmax. For each head h, let L_h in R^{B x T_q x T_k} be the scaled dot-product logits. Form centered normalized logits X_h = (L_h - mean(L_h)) / (std(L_h) + epsilon), with the mean and standard deviation computed over valid query-key entries; optionally stop gradients through these statistics to prevent the regularizer from cheating through scale changes. Compute a differentiable global maximum using smax_tau(X) = tau log sum_i exp(X_i/tau), where i ranges over valid logits and tau starts at 0.1 to 0.3. Set N_eff to the number of valid logits, or initially use T_k per query and aggregate the largest row statistic. During a warm-up period, estimate C by fitting observed maxima M_N to a log N + b log log N + c, initializing C with a and checking whether b is near -3a/4. If fitting is unstable, use C = 1 after per-head standardization. Calculate Y_h = (smax_tau(X_h) - C(log N_eff - 0.75 log log N_eff)) / C, then add R_h = softplus(Y_h - y_0)^2 with y_0 initially equal to 2. Pseudocode: L = q @ k.T / sqrt(d); X = center_and_scale(L); M = tau * logsumexp(X / tau); m = C * (log(Neff) - 0.75 * log(log(Neff))); Y = (M - m) / C; loss = task_loss + lambda * mean(softplus(Y - y0)^2). The mathematical quantities are the -3/4 log log N correction and the tail-motivated threshold; correlation length, C, N_eff, tau, lambda, and y_0 are empirical choices. First test on a 6-layer Transformer trained on WikiText-2 or a synthetic long-range-copy task, comparing vanilla attention, fixed logit clipping, and this regularizer at equal FLOPs. Measure NaN and overflow frequency, maximum-logit quantiles, gradient-norm spikes, validation perplexity, and attention entropy. Success means fewer extreme-logit or gradient events with equal or better perplexity, especially when evaluating at sequence lengths longer than those used in training.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.