One-sided BLO activation regularizer

(1) Apply this during training to pre-softmax attention logits or post-MLP residual activations. For each batch element, attention head, and sequence row, form overlapping token windows of length 32 or 64. For hidden states, use windows along the token dimension independently for each channel. Add alpha times the BLO penalty to the task loss. (2) Pseudocode: z = component_output; windows = unfold(z, size=L, stride=L//2); mean = windows.mean(dim=-1); m = -tau * logsumexp(-windows/tau, dim=-1); B = mean - m; reg = logsumexp(rho * B) / rho; loss = task_loss + alpha * reg; backpropagate normally. Use tau between 0.05 and 0.2 times the batch-wide standard deviation and rho between 5 and 20. (3) The theorem supplies the interpretation of B and the sharp exponential tail envelope; the discrete window maximum and soft minimum are engineering estimators and are not claimed to satisfy the continuous theorem exactly. Log the empirical exceedance curve p(lambda) = fraction of entries with z_t - m_W > lambda and compare it with e exp(-lambda/max_W B_W). (4) First experiment: train a six-layer, eight-head Transformer on WikiText-2 or a synthetic long-range copy task, comparing no regularizer, RMS activation regularization, and this BLO penalty at matched FLOPs. Success means lower 99th and 99.9th percentile logit magnitudes, fewer attention-softmax saturations, and equal or lower validation perplexity without slower loss descent.