# Sparse-Cost Temperature Calibration

- ID: 3001
- Canonical URL: https://synthcore.org/idea/3001/sparse-cost-temperature-calibration
- API JSON: https://synthcore.org/api/idea/3001.json
- API Markdown: https://synthcore.org/api/idea/3001.md
- Verification status: unverified
- Source: [arXiv:2609.01278](https://arxiv.org/abs/2609.01278)
- Category: training
- Solves: stability, accuracy, generalization
- ML areas: attention, transformer, moe-routing, scheduler
- Math tags: optimal-transport, statistics, linear-algebra, information-theory
- Ratings: usefulness 6/10; difficulty 4/10; novelty 8/10

## Idea description

Use a small set of known or trusted pairwise costs to estimate the effective entropic temperature of a Sinkhorn attention or mixture-of-experts routing layer directly from its observed transport plan. This provides a calibration controller that can detect over-concentrated routing and adjust epsilon without backpropagating through a costly temperature search.

## Mathematical statement

Let W be the observed transport plan, delta a positive numerical floor, and B = -H_m log(W plus delta) H_n. If a subset Omega of cost entries is known, the inverse-transport relation implies B_ij = C_centered,ij divided by epsilon, where C_centered = H_m C H_n. The least-squares temperature estimate minimizes the squared discrepancy between epsilon B_ij and C_centered,ij over Omega. Its closed form is epsilon_hat = sum over Omega of B_ij C_centered,ij divided by sum over Omega of B_ij squared. A normalized residual R measures whether the observed plan is consistent with the entropic model. A low R indicates reliable calibration; a high R indicates model mismatch, insufficient Sinkhorn convergence, or noisy cost references.

## Key formulas

- $$B=-H_m\log(W+\delta)H_n=\frac{H_mCH_n}{\varepsilon}$$
- $$\widehat{\varepsilon}=\frac{\sum_{(i,j)\in\Omega}B_{ij}C_{\mathrm{centered},ij}}{\sum_{(i,j)\in\Omega}B_{ij}^{2}},\qquad C_{\mathrm{centered}}=H_mCH_n$$
- $$R=\frac{\sum_{(i,j)\in\Omega}\left(\widehat{\varepsilon}B_{ij}-C_{\mathrm{centered},ij}\right)^2}{\sum_{(i,j)\in\Omega}C_{\mathrm{centered},ij}^{2}}$$
- $$\varepsilon_{t+1}=\operatorname{clip}\left(\varepsilon_t\left(\frac{\widehat{\varepsilon}_t}{\varepsilon_t}\right)^\alpha,\varepsilon_{\min},\varepsilon_{\max}\right),\qquad 0<\alpha\leq 1$$

## Implementation notes

Apply this controller to the temperature of a Sinkhorn attention head or an MoE routing matrix. Trusted costs can come from synthetic teacher geometry, distances between token metadata, class relations, or a small routing table; only a sparse index set Omega is required. Every T training steps, compute W, form B = -H_m log(W plus delta) H_n, gather B and centered reference costs on Omega, and calculate epsilon_hat using the closed-form regression formula. Compute R as the normalized residual. If R is below a threshold, update the live temperature with the damped geometric rule and clip it to safe limits. If R is high, freeze epsilon, increase Sinkhorn iterations, and log a model-mismatch warning rather than applying an unreliable update. The closed-form estimator and residual are the paper mechanism; damping, clipping, threshold, and update frequency are empirical design choices. First test on a small Transformer for synthetic sequence classification in which pairwise costs are Euclidean distances and 5 percent of costs are exposed. Compare fixed epsilon, gradient-learned epsilon, and inverse-OT calibration. The prediction is epsilon_hat divided by true epsilon within 10 percent when R is below 0.01. With independent observation noise, estimation error should decrease approximately as the inverse square root of the number of trusted entries. R should increase before routing entropy collapses when the learned attention no longer follows an entropic OT model.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.