Fuzzy permutation attention

Integrate the module into the attention-logit path. Given Q, K, and V in R^(n by d), precompute fuzzy basis matrices P_up_n_sigma for a small set K = union_{k=1}^{k_max} S_k, with k_max equal to 3 or 4. Do not materialize every embedding separately: enumerate subsets I and J and accumulate the integer counts into an n by n matrix for each basis, or precompute the matrices once for a fixed sequence length. At each forward pass, produce coefficient logits u_sigma from a pooled query/key statistic or from a learned per-head parameter, set alpha = softmax(u), and compute B = sum_sigma alpha_sigma P_up_n_sigma divided by sum_{i,j} B_{ij}. Add beta log(B + epsilon) to the usual scaled dot-product logits before row-wise softmax; use a learnable beta constrained to be nonnegative and initialize it near 0.1 so the baseline attention remains recoverable. Add lambda times L_bal to the training loss, where rho and J are defined above. For an exact variant, solve a small projected coefficient problem after each optimizer step: minimize the squared Frobenius norm of sum_sigma alpha_sigma P_up_n_sigma minus rho J, subject to alpha_sigma nonnegative and sum_sigma alpha_sigma equal to one, using five projected-gradient iterations. The paper supplies the embedding construction and exact balancing equation; the experiment learns alpha and beta. First test a 4-layer, 4-head Transformer with sequence length 32 on synthetic copy, reverse, sorting, and monotone sequence-to-sequence tasks. Compare standard attention, standard attention with an entropy or balance loss, and fuzzy attention at equal hidden size and FLOPs. Success means higher exact-match accuracy or faster loss descent on long-range copy and sorting, lower variance of attention row and column marginals, and the ability to replace some full-attention heads without accuracy loss.