# One-sided BLO activation regularizer

- ID: 2102
- Canonical URL: https://synthcore.org/idea/2102/one-sided-blo-activation-regularizer
- API JSON: https://synthcore.org/api/idea/2102.json
- API Markdown: https://synthcore.org/api/idea/2102.md
- Verification status: unverified
- Source: [arXiv:2608.10892](https://arxiv.org/abs/2608.10892)
- Category: regularization
- Solves: stability, generalization
- ML areas: transformer, attention, regularization
- Math tags: harmonic-analysis, probability, inequalities
- Ratings: usefulness 5/10; difficulty 4/10; novelty 8/10

## Idea description

Regularize hidden activations or attention logits by their local mean excess above the local minimum, rather than by symmetric variance or absolute magnitude. The penalty specifically suppresses upper-tail spikes while remaining invariant to adding a constant offset to every value in a local window.

## Mathematical statement

For an integrable real function f on an interval I0, the paper defines the bounded lower oscillation norm ||f||_{BLO(I0)} = sup_{I subseteq I0}(f_I - ess inf_I f), where f_I = |I|^{-1} integral_I f(x) dx. Its sharp one-sided John-Nirenberg inequality states that for every subinterval I subseteq I0 and lambda > 0, |{x in I: f(x) - ess inf_I f > lambda}|/|I| <= e exp(-lambda/B), with B = ||f||_{BLO(I0)}; the constants e and 1 are sharp. For a discrete activation vector z in R^T, approximate the essential infimum on a window W by a differentiable soft minimum m_W(z) = -tau log sum_{t in W} exp(-z_t/tau), and estimate the discrete BLO quantity by B_W = |W|^{-1} sum_{t in W} z_t - m_W(z). Penalize a smooth maximum of B_W over windows.

## Key formulas

- $$\|f\|_{\mathrm{BLO}(I_0)}=\sup_{I\subseteq I_0}\left(f_I-\operatorname*{ess\,inf}_{I}f\right),\qquad f_I=\frac{1}{|I|}\int_I f(x)\,dx.$$
- $$\frac{1}{|I|}\left|\left\{x\in I:f(x)-\operatorname*{ess\,inf}_{I}f>\lambda\right\}\right|\leq e\exp\left(-\frac{\lambda}{\|f\|_{\mathrm{BLO}(I_0)}}\right).$$
- $$m_W(z)=-\tau\log\sum_{t\in W}e^{-z_t/\tau},\qquad B_W(z)=\frac{1}{|W|}\sum_{t\in W}z_t-m_W(z),\qquad \mathcal L_{\mathrm{BLO}}=\frac{1}{\rho}\log\sum_W e^{\rho B_W(z)}.$$

## Implementation notes

(1) Apply this during training to pre-softmax attention logits or post-MLP residual activations. For each batch element, attention head, and sequence row, form overlapping token windows of length 32 or 64. For hidden states, use windows along the token dimension independently for each channel. Add alpha times the BLO penalty to the task loss. (2) Pseudocode: `z = component_output; windows = unfold(z, size=L, stride=L//2); mean = windows.mean(dim=-1); m = -tau * logsumexp(-windows/tau, dim=-1); B = mean - m; reg = logsumexp(rho * B) / rho; loss = task_loss + alpha * reg`; backpropagate normally. Use tau between 0.05 and 0.2 times the batch-wide standard deviation and rho between 5 and 20. (3) The theorem supplies the interpretation of B and the sharp exponential tail envelope; the discrete window maximum and soft minimum are engineering estimators and are not claimed to satisfy the continuous theorem exactly. Log the empirical exceedance curve p(lambda) = fraction of entries with z_t - m_W > lambda and compare it with e exp(-lambda/max_W B_W). (4) First experiment: train a six-layer, eight-head Transformer on WikiText-2 or a synthetic long-range copy task, comparing no regularizer, RMS activation regularization, and this BLO penalty at matched FLOPs. Success means lower 99th and 99.9th percentile logit magnitudes, fewer attention-softmax saturations, and equal or lower validation perplexity without slower loss descent.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.