# Stability-calibrated Sinkhorn attention

- ID: 2812
- Canonical URL: https://synthcore.org/idea/2812/stability-calibrated-sinkhorn-attention
- API JSON: https://synthcore.org/api/idea/2812.json
- API Markdown: https://synthcore.org/api/idea/2812.md
- Verification status: unverified
- Source: [arXiv:2608.29152](https://arxiv.org/abs/2608.29152)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: attention, moe-routing, regularization
- Math tags: optimal-transport, probability, statistics, convex-analysis
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace independently normalized attention or routing weights with an entropic doubly stochastic transport plan, while choosing its regularization ε using the paper's explicit statistical-stability bound. Increase ε when residual inversion or minibatch fluctuations are amplified, and decrease it only when the estimated bound permits sharper assignments.

## Mathematical statement

The paper's polynomial-stability principle states that if the summed entropy integral is bounded by A₀ ε^(-p) sqrt(Hε), the function-class envelope is bounded by E₀ ε^(-a), and the population Sinkhorn residual has inverse stability constant K₀ ε^(-r), then with probability at least 1−δ, the quotient sup-norm error is at most C K₀ / sqrt(n) times [A₀ ε^(1−r−p) sqrt(Hε) + E₀ ε^(1−r−a) sqrt(log(4/δ))]. Here n is the number of tokens or routing items, ε is the entropic temperature, Hε is the entropy-complexity term for normalized kernel sections, A₀, E₀, and C are scale constants, and a, p, and r describe envelope, complexity, and inverse-residual stability dependence on ε. The quotient metric d∞([u],[v]) = inf over b in R of ||u−v−b||∞ = one half of osc(u−v) removes the arbitrary additive gauge of Sinkhorn potentials. Use the bound as a temperature controller, with K₀ ε^(-r) estimated by the local inverse sensitivity of the Sinkhorn residual.

## Key formulas

- $$d_{\infty}([\widehat{f}_{\varepsilon,n}],[f_{\varepsilon}])\leq\frac{CK_{0}}{\sqrt{n}}\left[A_{0}\varepsilon^{1-r-p}\sqrt{H_{\varepsilon}}+E_{0}\varepsilon^{1-r-a}\sqrt{\log(4/\delta)}\right].$$
- $$d_{\infty}([u],[v])=\inf_{b\in\mathbb R}\|u-v-b\|_{\infty}=\tfrac12\operatorname{osc}(u-v).$$
- $$P_{ij}(\varepsilon)=\exp\!\left(\frac{q_i^{\top}k_j+\alpha_i+\beta_j}{\varepsilon}\right),\qquad \sum_jP_{ij}=r_i,\quad \sum_iP_{ij}=c_j.$$
- $$\widehat B(\varepsilon)=\frac{\widehat K(\varepsilon)}{\sqrt n}\left[A_0\varepsilon^{1-r-p}\sqrt{\widehat H_\varepsilon}+E_0\varepsilon^{1-r-a}\sqrt{\log(4/\delta)}\right],\qquad \varepsilon_{t+1}=\operatorname{clip}\bigl(\varepsilon_t\exp(-\eta\,\operatorname{sign}(\widehat B_t-\tau)),\varepsilon_{\min},\varepsilon_{\max}\bigr).$$

## Implementation notes

Integrate this into the attention-logit or MoE-router path before the final weighted aggregation. For a minibatch of n tokens, compute scores Sij = qiᵀkj, define Kij = exp(Sij/ε), set row marginals ri = 1/n and column marginals cj = 1/m, and run 5–20 log-domain Sinkhorn iterations to obtain potentials α and β and transport weights P. Center every potential by subtracting its median to fix the quotient gauge. Maintain an exponential moving average of the observed Sinkhorn residual R = (P1−r, Pᵀ1−c). Estimate the local inverse-stability factor K-hat by finite differences: perturb the centered potentials or logits by small random vectors ξ, recompute the residual, and use K-hat = max ||ξ||∞ / ||ΔR||∞ over several probes, with damping and clipping. Estimate H-hatε from empirical variance or a simple covering proxy of normalized kernel rows; initially use a constant calibrated on a held-out batch. Fit exponents a, p, and r by regressing the envelope, complexity proxy, and K-hat against log ε over a temperature grid. Compute B-hat(ε) from the displayed bound and update ε toward a target noise budget τ. The first experiment should use CIFAR-10 or a small language model, comparing softmax attention, fixed-temperature Sinkhorn attention, and calibrated Sinkhorn attention at equal FLOPs. Measure validation loss, attention-plan variance, entropy, residual size, and loss spikes under small minibatches. Success means lower plan variance and equal-or-better validation loss without more Sinkhorn iterations.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.