Spectral-gap regularized doubly stochastic attention

Insert the regularizer in a Sinkhorn-based attention layer during training. For each batch element, form A=exp((S-c)/tau), run a detached or low-iteration Sinkhorn pass to obtain A_star=diag(u)A diag(v), and define B=P^{-1/2}A_star Q^{-1}A_star^T P^{-1/2}. For square self-attention with uniform marginals p=q=1/L, P and Q are scalar multiples of the identity; for general marginals use the supplied diagonal vectors. Estimate the dominant eigenvector w with 5 power iterations on B, normalize it, then run 3-5 projected power iterations z <- (I-ww^T)Bz followed by normalization. Estimate lambda2=z^T Bz and add L_gap=[lambda2-lambda_target]_+^2 to the task loss. Set lambda_target initially to 0.7 or to the median lambda2 observed during a warm-up epoch; use beta between 1e-4 and 1e-2 so the regularizer cannot dominate language-model loss. Compute the marginal residual r and optionally include gamma||r||^2, but do not backpropagate through more Sinkhorn iterations than the baseline. The eigenvalue estimate and regularizer come from the paper's contraction formula; lambda_target, beta, and power-iteration count are empirical hyperparameters. First test on a 2-layer, 4-head Transformer on a synthetic permutation/assignment dataset and WikiText-2, comparing ordinary Sinkhorn attention with the regularized version at the same fixed iteration budget. Track estimated lambda2, actual residual decay per iteration, task accuracy or perplexity, gradient stability, and wall-clock overhead. The hypothesis is that training lowers lambda2, produces steeper residual decay, and permits reducing fixed Sinkhorn iterations by 25-50% without increasing marginal error or validation loss.