Cofactor-Stable Attention

Integrate this into the attention-logit path of a Transformer encoder or decoder, initially on one or two heads per layer. Given logits Z with shape batch by heads by sequence length by sequence length, apply the usual row-wise softmax to obtain A, construct L = I - A, and compute one cofactor vector for every selected batch-head matrix. For a first MVP with sequence length at most 32, explicitly form each principal minor: for every token i, delete row i and column i from L and compute its determinant with a differentiable linear-algebra routine. Use float64 for this diagnostic branch, or use slogdet and reconstruct c_i from sign times exp(log absolute determinant). Add epsilon = 1e-8. Normalize positive cofactors into pi, compute the margin penalty negative-log of the absolute cofactor sum, and optionally add beta times KL divergence from the uniform distribution. Add the regularizer to the task loss with alpha between 1e-4 and 1e-3. In pseudocode: A = softmax(Z); L = I - A; for each i, c[i] = det(L with row i and column i removed); pi = normalize(relu(c) + epsilon); regularizer = -log(abs(sum(c)) + epsilon) + beta times KL(uniform, pi); total loss = task loss + alpha times regularizer. Backpropagate through the determinant values, but avoid gradients through sign, clipping, or numerical fallback decisions. For longer sequences, approximate pi using power iteration on A transpose, while evaluating exact cofactors on short calibration sequences. Test a small six-layer Transformer on WikiText-2 or a patch-token classification task against baseline attention at equal FLOPs. Log validation loss, gradient-norm variance, the minimum cofactor probability, the number of near-zero cofactors, and the attention spectral gap 1 minus the magnitude of the second eigenvalue of A. Success means fewer nearly disconnected heads, lower gradient spikes, and equal or better validation accuracy.