Overlap-Gap Temperature Controller

Integrate the method into the attention-logit path during training, immediately before the row-wise softmax. For each batch and attention head, compute the usual scaled similarity matrix $S=QK^T/\sqrt d$. Estimate clusters using one or two k-means iterations on detached query vectors, or assign tokens to the nearest of $k$ learned centroids; begin with $k=\min(8,N)$. For every query row $i$, compute the largest similarity to tokens with the same cluster assignment and the largest similarity to tokens with a different assignment. Their difference is the empirical $\Delta_i$. Treat the learned or fixed logit scale as the current $\beta$, choose a target cross-cluster mass $\tau=0.05$, and calculate $\beta_i^{\max}$ using the displayed formula with $\varepsilon=10^{-3}$. Use $\beta_i=\operatorname{clip}(\beta,\beta_{\min},\operatorname{median}_i\beta_i^{\max})$, with an exponential moving average of the median at decay 0.95. Stop gradients through cluster assignments and gap estimates initially. The paper-derived quantity is the exponential suppression bound; cluster membership and gap values are estimated from each batch. First test a 6-layer, 8-head Transformer on WikiText-2 against fixed scaling and standard learned temperature at equal FLOPs. Record validation loss, attention entropy, effective rank of token states, cross-cluster attention mass, gradient norms, and failed runs. Success is at least 20% fewer instability events or lower validation loss at high learning rates without added inference cost.