Entropy-selected simplex parameters

(1) Integration point: apply the regularized update to a learned distribution over MoE experts, attention heads, retrieval items, diffusion mixture components, or token-selection candidates. Keep the ordinary neural-network parameters trained by AdamW, but update the simplex parameter \(p\) with the KL-prox rule; alternatively add \(\log p\) as a bias to logits and periodically update that bias. Choose \(r\) uniform for anti-collapse, pretrained \(p\) for conservative fine-tuning, or an exponential moving average of observed usage for adaptive balancing. (2) Pseudocode: initialize \(p=r\) or uniform with floor \(10^{-8}\); for each minibatch compute the vector cost \(c=\texttt{cost}(p)\); choose \(\lambda_t=\lambda_0\max(0,1-t/T)\); calculate \(u_i=\log p_i-\eta[c_i+\lambda_t\log(p_i/r_i)]\); set \(p=\operatorname{softmax}(u)\) or normalize \(\exp(u)\); use \(p\) in the gate or attention computation. For a predictor-corrector version, evaluate \(c\) at the intermediate distribution before the final update. (3) The KL term, its logarithmic gradient, and multiplicative interpolation toward \(r\) are computed analytically. Estimate the task cost from minibatch loss and the collapse statistic; tune \(\lambda_0\) and the annealing horizon on validation data. Monitor minimum probability, entropy, effective number of experts \(\exp(H(p))\), and task loss. (4) First cheap experiment: fine-tune a small Transformer with 8 attention heads while learning a head-mixture distribution on a subset of GLUE, or train a 4-expert CIFAR-100 MoE. Compare no regularization, standard L2-on-logits, entropy maximization, and KL-to-uniform. Success is fewer dead heads or experts, lower run-to-run variance, and equal or better validation accuracy; report the Pareto curve of accuracy versus effective support and test whether annealing reaches better final specialization than a fixed entropy coefficient.