Sharp BLO-based adaptive spike clipping

(1) Insert the operation immediately before attention softmax, before an activation quantizer, or before a tensor is written to a KV cache. Use non-overlapping or half-overlapping windows over the sequence dimension. For attention logits, compute one threshold per query row and head; for hidden states, compute one threshold per channel and token window. (2) Pseudocode: z = logits_or_activation; m = amin(z, dim=window); B = mean(z, dim=window) - m; B = maximum(B, eps); threshold = m + B * log(e / delta); z_clip = minimum(z, threshold). Initially apply stop-gradient to m and B so the model cannot reduce the penalty by manipulating its threshold. At inference, maintain an exponential moving average B_ema = beta * B_ema + (1-beta) * B and use B_ema. (3) The paper determines the factor log(e/delta); the window choice, tensor-wise B estimate, and any safety multiplier c are empirical. On a held-out calibration set, choose the smallest c >= 1 for which the observed clipped fraction is at most delta, then evaluate on a separate test set. (4) First experiment: run a pretrained small Transformer on WikiText-2 or C4 with attention-logit clipping at delta = 1e-2 and 1e-3, comparing fixed-value clipping and percentile clipping. Measure attention entropy, FP16/FP8 overflow or underflow, perplexity, and latency. Success is fewer saturated or numerically invalid rows and improved reduced-precision stability at unchanged or nearly unchanged perplexity. For KV-cache activations, also measure dynamic range and INT8 quantization error at the same bit width.