Higher-Order Coactivation Envelope

Integrate the method at the attention-affinity or MoE-router level. For attention, compute A = sigmoid(QK^T divided by sqrt(d)) on selected sequence blocks, then symmetrize it as W = (A + A^T)/2. For MoE, define W_ij = sum_e p_ie p_je, where p_ie is token i's probability of selecting expert e. Start with s = 3 and r = 4. Before training, construct the envelope numerically: for every integer branch u from r through a chosen maximum U, evaluate q_s^(u)(a) and q_r^(u)(a) on a dense grid for a in [0, 1/(u-1)], retain the minimum valid q_r at each q_s, and store a monotone interpolant F. At each step, sample M tuples of distinct tokens, compute the Monte Carlo estimates of g_s and g_r using the displayed product formula, interpolate F(g_s), and add lambda times the squared hinge violation to the task loss. Warm lambda from zero, use independent samples for the two estimates or stop-gradient through F(g_s) to reduce noisy feedback, and average across heads or routing blocks. The first experiment should use a small six-layer transformer on WikiText-2, comparing the regularizer with ordinary attention at equal FLOPs and parameter count. Record validation perplexity, attention entropy, the envelope violation, and batch-to-batch variance of these statistics. For MoE, also measure expert-load variance. Success means lower variance or better perplexity at equal compute without attention collapse or increased routing imbalance.