Polar Slack Attention

Integrate the construction into one attention layer or an MoE router whose sequence positions or experts are indexed by N geometric slots. Construct a spherical-design matrix X with N columns in R^d and normalize every column to unit norm. Store positive support values h_i using h_i = exp(g_i), where g_i is fixed initially or is a trainable scalar. Store U as an orthogonal matrix, initially U = I; if it is trained, parameterize it with a Cayley transform or periodically project the raw matrix onto O(d) using QR. At initialization compute rho = max_{i,j} x_i^T U^T x_j and choose c = (1 + margin) rho divided by min_{i,j}(h_i h_j), with margin such as 0.01; if rho is nonpositive, use c = 1. Recompute A = c outer(h,h) - X^T U^T X on every forward pass for small N, or cache it and refresh it periodically for larger N. Clamp A to max(A,0), form B = log(A + epsilon), and add alpha B to the standard QK^T divided by sqrt(d_k). For sparse inference, keep only the k largest A entries in each row or retain entries with A_{ij} greater than a threshold tau, then apply a sparse softmax over the retained indices. The paper-derived quantities are the factorized slack formula, its nonnegativity target, and its rank structure; alpha, epsilon, k, tau, and whether h and U are trainable are empirical choices. First run a small experiment with a 2-layer, 4-head Transformer on CIFAR-10 converted to patches or a character-level language model on Tiny Shakespeare. Compare dense attention, ordinary top-k attention, and geometric slack attention at equal hidden size. Measure validation loss, active edges per row, attention FLOPs, GPU memory, and seed-to-seed variance. The method succeeds if it reaches the same validation loss with at least 2x fewer retained attention edges, or gives faster loss reduction at equal FLOPs without producing negative-kernel numerical failures.