Core-Halo Attention and Routing Regularizer

Integrate the mechanism at the probability-output level rather than modifying Adam or another parameter optimizer. For a Transformer, apply it independently to every attention row after the softmax. For an MoE model, apply it to each token's gate vector. For an RL policy, apply it to the action probabilities. Given logits z and temperature T, compute p = softmax(z/T), then calculate H, E, and C and minimize L_total = L_task - gamma times mean(C), where gamma is nonnegative and is ramped from zero. Pseudocode is: p = softmax(logits / T); H = -(p * log(p + eps)).sum(dim=-1); E = (p * p).sum(dim=-1); C = (H / log(n)) * ((n * E - 1) / (n - 1)); loss = task_loss - gamma * C.mean(); loss.backward(). Use ordinary gradients through C initially; if gradient feedback destabilizes training, compare against a stop-gradient version that computes C only as a routing target. The paper provides the two-level stationary equation and the global prediction that the smallest core is optimal. The implementation should estimate the empirical core multiplicity by sorting each probability vector, fitting the best two-level approximation for every m from 1 to n - 1, and recording the m with lowest squared fitting error or highest measured C_m. First run a cheap experiment using a small two-layer Transformer on WikiText-2, comparing no regularizer, ordinary entropy regularization, and the core-halo regularizer at matched coefficient magnitudes. Also run a direct probability-vector optimization with the task loss removed, using random logits and several n values. The primary prediction is structural: rows dominated by the regularizer should fit two plateaus, with within-core and within-halo variance much smaller than the squared gap between their means. In the direct optimization test, the best fitted multiplicity should be m = 1 in at least 80 percent of random restarts, and the achieved C should approach the numerically optimized one-core value. In the neural experiment, increasing gamma should produce a transition from nearly uniform rows to two-level rows; record the gamma at which the two-level fit becomes dominant and test whether this transition shifts approximately inversely with the task-gradient scale.