# Sharp BLO-based adaptive spike clipping

- ID: 2103
- Canonical URL: https://synthcore.org/idea/2103/sharp-blo-based-adaptive-spike-clipping
- API JSON: https://synthcore.org/api/idea/2103.json
- API Markdown: https://synthcore.org/api/idea/2103.md
- Verification status: unverified
- Source: [arXiv:2608.10892](https://arxiv.org/abs/2608.10892)
- Category: memory
- Solves: stability, memory
- ML areas: attention, transformer, quantization, kv-cache
- Math tags: harmonic-analysis, probability, inequalities, numerical-analysis
- Ratings: usefulness 5/10; difficulty 3/10; novelty 7/10

## Idea description

Use the sharp exponential tail bound to set a local clipping threshold from a desired exceedance probability. Instead of globally clipping activations at a fixed value or percentile, clip each local window at its minimum plus B log(e/delta), where delta is the tolerated fraction of clipped entries.

## Mathematical statement

Let B = ||f||_{BLO(I0)}, and define the local excess u(x) = f(x) - ess inf_I f on an interval I. The paper's sharp inequality gives Pr_I[u > lambda] <= e exp(-lambda/B). Solving this inequality for a target exceedance probability delta in (0,1) yields lambda_delta = B log(e/delta). In a tensor window W, estimate the essential infimum by the exact minimum m_W = min_{t in W} z_t and estimate B by B_W = mean(z_W) - m_W. The adapted clipping rule is z'_t = min(z_t, m_W + B_W log(e/delta)). The theorem assumes a true BLO bound, so its tensor application is a calibrated heuristic; the testable prediction is that the empirical clipped fraction is near or below delta after conservative calibration.

## Key formulas

- $$\frac{1}{|I|}\left|\left\{x\in I:f(x)-\operatorname*{ess\,inf}_{I}f>\lambda\right\}\right|\leq e\exp\left(-\frac{\lambda}{B}\right),\qquad B=\|f\|_{\mathrm{BLO}(I_0)}.$$
- $$\lambda_\delta=B\log\frac{e}{\delta},\qquad e\exp\left(-\frac{\lambda_\delta}{B}\right)=\delta.$$
- $$m_W=\min_{t\in W}z_t,\qquad B_W=\frac{1}{|W|}\sum_{t\in W}z_t-m_W,\qquad z'_t=\min\left(z_t,\,m_W+B_W\log\frac{e}{\delta}\right).$$

## Implementation notes

(1) Insert the operation immediately before attention softmax, before an activation quantizer, or before a tensor is written to a KV cache. Use non-overlapping or half-overlapping windows over the sequence dimension. For attention logits, compute one threshold per query row and head; for hidden states, compute one threshold per channel and token window. (2) Pseudocode: `z = logits_or_activation; m = amin(z, dim=window); B = mean(z, dim=window) - m; B = maximum(B, eps); threshold = m + B * log(e / delta); z_clip = minimum(z, threshold)`. Initially apply stop-gradient to m and B so the model cannot reduce the penalty by manipulating its threshold. At inference, maintain an exponential moving average `B_ema = beta * B_ema + (1-beta) * B` and use B_ema. (3) The paper determines the factor log(e/delta); the window choice, tensor-wise B estimate, and any safety multiplier c are empirical. On a held-out calibration set, choose the smallest c >= 1 for which the observed clipped fraction is at most delta, then evaluate on a separate test set. (4) First experiment: run a pretrained small Transformer on WikiText-2 or C4 with attention-logit clipping at delta = 1e-2 and 1e-3, comparing fixed-value clipping and percentile clipping. Measure attention entropy, FP16/FP8 overflow or underflow, perplexity, and latency. Success is fewer saturated or numerically invalid rows and improved reduced-precision stability at unchanged or nearly unchanged perplexity. For KV-cache activations, also measure dynamic range and INT8 quantization error at the same bit width.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.