Mean-Scaled Tail Retention Controller

(1) Integration point: insert this controller before the minibatch loss reduction. For each example, compute a score $s_i$, initially the per-example gradient norm using a cheap last-layer gradient or a detached loss-gradient proxy; for activation control, use the mean absolute activation of a selected layer. Maintain an exponential moving estimate of the score mean, $\hat{s}_t=(1-\beta)\hat{s}_{t-1}+\beta\,\mathrm{mean}(s_i)$, and set $b_t=\xi\hat{s}_t$. (2) Pseudocode: compute scores; set mask $m_i=\mathbf{1}[s_i\le b_t]$; let $r=\max(\mathrm{mean}(m_i),r_{\min})$; form the batch loss $L=\sum_i m_i\ell_i/(nr)$ for importance-corrected training, or $L=\sum_i m_i\ell_i/n$ for intentional tail regularization; backpropagate and update parameters. Log $\hat{s}_t$, $b_t$, retained fraction $r$, and the empirical map from $\hat{s}_t$ to the next-round retained mean. (3) Computed from the paper's mechanism: conditional truncation, normalization by retained mass, and the fixed-point criterion. Estimated empirically: score density, $F_t(b_t)$, $T'(\bar{s}_*)$ using finite differences, and the effect of score variance. (4) First cheap experiment: train a small 2-layer MLP on MNIST or CIFAR-10 with SGD, comparing vanilla SGD, global gradient clipping, and this controller for $\xi\in\{0.8,1.0,1.5,2.0,3.0\}$. Use both corrected and uncorrected variants. The predicted signature is a sharp change in retained fraction when $b_t$ crosses the score-distribution shoulder, convergence of $\hat{s}_t$ to a stationary value when the measured local slope $|T'|<1$, and oscillation or collapse when $|T'|>1$. Across runs, increasing score variance at fixed mean should reduce the retained fraction and increase training-loss bias; the measured fixed point should be reproducible within approximately 20% across seeds.