q-Ary Influence Overlap Regularizer

Apply the regularizer to sparse attention supports, MoE routing supports, or GNN neighborhoods. For each minibatch and head or expert h, obtain a support S_h from the actual routing probabilities. With Bernoulli or Gumbel-top-k sampling, draw a second independent support \widetilde S_h using fresh random noise. Compute overlap_count=(S_h and \widetilde S_h).sum(-1), then evaluate q^{overlap_count}=exp(log(q)*overlap_count) and average across tokens, heads, and M=2--8 resamples. Add this quantity minus one to the task loss, using a warm-up schedule for lambda. If numerical overflow occurs, clip overlap_count or calculate a log-moment. A straight-through estimator can be used for discrete top-k masks; the masks themselves remain the true masks used by the forward pass. The exact paper-derived statistic is the empirical estimator of E[q^{|S cap S'|}]-1. The uniform categorical reference distribution need not be sampled during training, although it can be used to evaluate calibration. First train a 12-layer, 8-head Transformer on WikiText-2 with top-8 sparse attention. Compare no regularizer, standard Jaccard diversity, and this q-ary moment for q in {2,4,8}. Measure validation perplexity, support-collision rate, gradient-noise scale, and performance after masking 10% of tokens. Success means lower severe-overlap frequency and gradient variance at equal sparsity, with unchanged or improved perplexity; an ablation should show that q-dependent weighting beats ordinary Jaccard for larger categorical dimensions.