# Gauge-Free Inverse OT Attention

- ID: 3000
- Canonical URL: https://synthcore.org/idea/3000/gauge-free-inverse-ot-attention
- API JSON: https://synthcore.org/api/idea/3000.json
- API Markdown: https://synthcore.org/api/idea/3000.md
- Verification status: queued_mechanism
- Source: [arXiv:2609.01278](https://arxiv.org/abs/2609.01278)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: attention, transformer, regularization
- Math tags: optimal-transport, linear-algebra, information-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace ordinary row-softmax attention with a doubly stochastic Sinkhorn attention plan W, and periodically recover a gauge-fixed pairwise cost from W using the exact double-centering identity. Use this recovered cost to initialize or regularize a structured attention score, making the attention geometry identifiable despite arbitrary query and key row and column offsets.

## Mathematical statement

For source and target marginals s and r, entropic optimal transport has coupling W = diag(u) K diag(v), where K_ij = exp(-C_ij divided by epsilon), C is the cost matrix, epsilon is the entropic temperature, and u and v are Sinkhorn scaling vectors. Consequently, log W = log(u) 1-transpose + 1 log(v)-transpose - C divided by epsilon. Let H_m = I_m - 1 1-transpose divided by m and H_n = I_n - 1 1-transpose divided by n be centering matrices. Double-centering removes the row and column gauge terms, yielding H_m log(W) H_n = -H_m C H_n divided by epsilon. The recoverable gauge-fixed cost is therefore C_hat = -epsilon H_m log(W plus delta) H_n, where delta is a positive numerical floor. Costs differing by a 1-transpose plus 1 b-transpose generate the same transport plan, so only the double-centered cost is identifiable.

## Key formulas

- $$W=\operatorname{diag}(u)K\operatorname{diag}(v),\qquad K_{ij}=\exp\left(-\frac{C_{ij}}{\varepsilon}\right)$$
- $$\log W=\log(u)\mathbf{1}^{\top}+\mathbf{1}\log(v)^{\top}-\frac{C}{\varepsilon}$$
- $$H_m(\log W)H_n=-\frac{1}{\varepsilon}H_mCH_n,\qquad H_q=I_q-\frac{1}{q}\mathbf{1}\mathbf{1}^{\top}$$
- $$\widehat C=-\varepsilon H_m\log(W+\delta)H_n$$

## Implementation notes

Integrate this into a cross-attention block. Convert attention scores S into a positive kernel K with K_ij = exp(-S_ij divided by epsilon), then run 5 to 20 log-domain Sinkhorn iterations to obtain W with prescribed source and target marginals. Keep value aggregation Y = W V, but add an inverse-cost consistency term or use the recovered cost as a structured bias for the next attention block. The pseudocode is: compute stabilized K; initialize positive u and v; repeat u = s divided elementwise by K v and v = r divided elementwise by K-transpose u; form W = diag(u) K diag(v); set L = log(W plus delta); compute C_inv = -epsilon H_m L H_n; add lambda times the squared Frobenius norm of H_m(S - C_inv)H_n to the training loss. The exact inverse identity is supplied by the paper. The engineer must estimate the effect of finite Sinkhorn convergence, floating-point flooring, and model mismatch. First run a synthetic test with random C, prescribed marginals, and generated W before training a two-layer Transformer on a small copy or patch-classification task. Compare softmax attention with Sinkhorn attention with and without the inverse-cost penalty. The prediction is that the relative centered reconstruction error will remain below 5 percent when the smallest W entry exceeds 10^-5 and Sinkhorn marginal error is below 10^-4. Adding any row-plus-column gauge to C must not change C_inv. Reconstruction error should increase sharply when W entries fall below delta.

## Verification

- Status: queued_mechanism
- Mechanism evidence: no
- Mechanism confirmed: no

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.