Targeted Information-Variance Regularization

Apply the regularizer to logits from a classifier head, the final token distribution of a small language model, or the gate probabilities of an MoE layer. Do not apply it to every layer initially; start with one output or routing layer so the added computation is negligible and task gradients remain dominant. During a warm-up lasting 5 to 10 percent of training, set lambda to zero and record the minibatch complexity. Choose Cstar as the median or 75th percentile of warm-up values, excluding the first unstable batches. Then execute: p = softmax(logits / temperature); q = clamp(p, 1e−8, 1); H = −sum(q times log(q), last dimension); C = sum(q times (−log(q) − H[..., None]) squared, last dimension); loss = task_loss + lambda times (mean(C) − Cstar) squared. C is computed exactly from the current softmax, while Cstar is estimated from warm-up data. Run a cheap experiment with a two-layer MLP on MNIST under label noise, comparing this method against no regularizer, entropy maximization, and label smoothing. Sweep lambda over 1e−4, 1e−3, and 1e−2. The quantitative prediction is a U-shaped response as Cstar varies: targets near zero permit collapse or uniformity, intermediate targets preserve nontrivial confidence structure, and excessively large targets destabilize logits. Measure maximum predicted probability, entropy, calibration error, validation accuracy, and final C. The method is supported only if C tracks Cstar within roughly 10 percent, reduces collapse, and outperforms entropy control at matched accuracy.