# Cofactor-Stable Attention

- ID: 379
- Canonical URL: https://synthcore.org/idea/379/cofactor-stable-attention
- API JSON: https://synthcore.org/api/idea/379.json
- API Markdown: https://synthcore.org/api/idea/379.md
- Verification status: unverified
- Source: [arXiv:2607.02843](https://arxiv.org/abs/2607.02843)
- Category: regularization
- Solves: stability, accuracy
- ML areas: attention, transformer, regularization
- Math tags: linear-algebra, spectral-theory, graph-theory
- Ratings: usefulness 5/10; difficulty 6/10; novelty 7/10

## Idea description

Treat each directed attention matrix as a graph transition matrix and form its Laplacian L = I - A. Compute the principal-cofactor vector to identify tokens with weak global access to the rest of the layer, and regularize the nonzero-eigenvalue product so attention does not become reducible or nearly singular. This targets pathological attention heads that isolate token groups and produce unstable or poorly propagated representations.

## Mathematical statement

The construction applies to any n by n matrix L satisfying L times the all-ones vector equals zero and rank(L) = n - 1. For index i, let L with row i and column i deleted be the principal minor, and define c_i as its determinant. The cofactor vector c satisfies c transpose times L equals zero. If the eigenvalues are 0, lambda_2, ..., lambda_n, then the stability margin m equals the sum of all principal cofactors and also equals the product of the nonzero eigenvalues: m = product from k = 2 to n of lambda_k. For an attention matrix A with A_ij = exp(z_ij) divided by the row sum of exp(z_ir), A is row-stochastic, so A times the all-ones vector equals the all-ones vector. Therefore L = I - A has a guaranteed zero eigenvalue. For an irreducible stochastic A, the cofactors are nonnegative and proportional to the stationary left eigenvector. Define pi_i = c_i divided by the sum of positive cofactors, and use the negative logarithm of the margin as a regularization term.

## Key formulas

- $$A_{ij}=\frac{\exp(z_{ij})}{\sum_{r=1}^{n}\exp(z_{ir})},\qquad A\mathbf{1}=\mathbf{1},\qquad L=I-A$$
- $$c_i=\det\left(L_{\widehat{i}\widehat{i}}\right),\qquad c=(c_1,\ldots,c_n)^\top,\qquad c^\top L=0$$
- $$m=\sum_{i=1}^{n}c_i=\prod_{k=2}^{n}\lambda_k(L),\qquad \lambda_1(L)=0$$
- $$\mathcal{L}_{\mathrm{cofactor}}=-\log\left(\left|\sum_{i=1}^{n}c_i\right|+\epsilon\right)+\beta\,\mathrm{KL}\left(u\,\middle\|\,\pi\right),\qquad \pi_i=\frac{\max(c_i,0)+\epsilon}{\sum_j(\max(c_j,0)+\epsilon)},\quad u_i=\frac{1}{n}$$

## Implementation notes

Integrate this into the attention-logit path of a Transformer encoder or decoder, initially on one or two heads per layer. Given logits Z with shape batch by heads by sequence length by sequence length, apply the usual row-wise softmax to obtain A, construct L = I - A, and compute one cofactor vector for every selected batch-head matrix. For a first MVP with sequence length at most 32, explicitly form each principal minor: for every token i, delete row i and column i from L and compute its determinant with a differentiable linear-algebra routine. Use float64 for this diagnostic branch, or use slogdet and reconstruct c_i from sign times exp(log absolute determinant). Add epsilon = 1e-8. Normalize positive cofactors into pi, compute the margin penalty negative-log of the absolute cofactor sum, and optionally add beta times KL divergence from the uniform distribution. Add the regularizer to the task loss with alpha between 1e-4 and 1e-3. In pseudocode: A = softmax(Z); L = I - A; for each i, c[i] = det(L with row i and column i removed); pi = normalize(relu(c) + epsilon); regularizer = -log(abs(sum(c)) + epsilon) + beta times KL(uniform, pi); total loss = task loss + alpha times regularizer. Backpropagate through the determinant values, but avoid gradients through sign, clipping, or numerical fallback decisions. For longer sequences, approximate pi using power iteration on A transpose, while evaluating exact cofactors on short calibration sequences. Test a small six-layer Transformer on WikiText-2 or a patch-token classification task against baseline attention at equal FLOPs. Log validation loss, gradient-norm variance, the minimum cofactor probability, the number of near-zero cofactors, and the attention spectral gap 1 minus the magnitude of the second eigenvalue of A. Success means fewer nearly disconnected heads, lower gradient spikes, and equal or better validation accuracy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.