# Spectral-Certified Sinkhorn Optimizer

- ID: 2228
- Canonical URL: https://synthcore.org/idea/2228/spectral-certified-sinkhorn-optimizer
- API JSON: https://synthcore.org/api/idea/2228.json
- API Markdown: https://synthcore.org/api/idea/2228.md
- Verification status: unverified
- Source: [arXiv:2608.13201](https://arxiv.org/abs/2608.13201)
- Category: optimization
- Solves: stability, speedup, accuracy
- ML areas: optimizer, attention, training-dynamics
- Math tags: optimal-transport, optimization, convex-analysis, spectral-theory, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Use the OT spectral bound as a conditioning signal for optimizing parameters of a neural cost or inverse-OT objective. Adapt the parameter step size and add a covariance floor whenever the estimated Jacobian lower bound collapses, preventing optimization from entering regions where Sinkhorn outputs become insensitive to the learned cost.

## Mathematical statement

Let $Q_{\theta}$ denote the observed conditional transition operator induced by the entropic OT plan, and let $\mathcal{J}_{\theta}=\partial Q_{\theta}/\partial\theta$ be its parameter Jacobian. The paper's core estimate is $\sigma_{\min}(\mathcal{J}_{\theta})\geq\gamma_{\theta}$ with $\gamma_{\theta}=\pi_{\min}\sqrt{\lambda_{\min}(\Sigma)}/(a_{\max}\varepsilon)$. For a cross-entropy objective built from the model operator, the paper derives local strong convexity $\mu\geq\pi_{\min}^{2}\lambda_{\min}(\Sigma)/\varepsilon^{2}=a_{\max}^{2}\gamma_{\theta}^{2}$. Here $\varepsilon>0$ is entropic regularization, $\pi_{\min}$ is the smallest plan entry, $\Sigma$ is feature covariance, $a_{\max}$ is the largest source marginal, and $\lambda_{\min}(\Sigma)$ measures feature identifiability. The paper also gives $L\leq\varepsilon\|\Phi^\top S_a\|_{\mathrm{op}}/[\pi_{\min}\lambda_{\min}(\Sigma)]$, where $\Phi$ is the feature matrix and $S_a$ is the source-marginal weighting operator. The adaptation uses $\gamma_{\theta}$ to scale gradients and uses an empirical Hessian estimate for a safeguarded step size.

## Key formulas

- $$\sigma_{\min}(\mathcal{J}_{\theta})\geq\gamma_{\theta}:=\frac{\pi_{\min}}{a_{\max}\varepsilon}\sqrt{\lambda_{\min}(\Sigma)}.$$
- $$\mu\geq\frac{\pi_{\min}^{2}\lambda_{\min}(\Sigma)}{\varepsilon^{2}}=a_{\max}^{2}\gamma_{\theta}^{2}.$$
- $$L\leq\frac{\varepsilon\|\Phi^{\top}S_a\|_{\mathrm{op}}}{\pi_{\min}\lambda_{\min}(\Sigma)}.$$
- $$\theta_{t+1}=\theta_t-\eta_t\frac{g_t}{\max(\widehat\gamma_t,\gamma_0)},\qquad \eta_t=\frac{\eta_0}{\max(\widehat L_t,L_0)},\qquad \widehat\gamma_t=\frac{\widehat\pi_{\min,t}\sqrt{\widehat\lambda_{\min}(\Sigma_t)}}{\widehat a_{\max,t}\varepsilon}.$$

## Implementation notes

(1) Integration point: modify the optimizer update for trainable cost parameters in a Sinkhorn layer, inverse-OT head, or OT-based attention module. Keep ordinary Adam or SGD for the rest of the network and apply the spectral rule only to $\theta$. (2) For each minibatch, compute the Sinkhorn plan $\pi_t$, pair-feature matrix $\Phi_t$, and covariance EMA $\Sigma_t=\beta\Sigma_{t-1}+(1-\beta)\Phi_t^\top\Phi_t/K^2$. Estimate $\widehat\pi_{\min,t}$ with the 1st percentile of plan entries to avoid a single numerical outlier, set $\widehat a_{\max,t}=\max_i a_i$, and compute $\widehat\lambda_{\min}(\Sigma_t)$ using an eigensolver for small feature dimension or Lanczos for large dimension. (3) Form $\widehat\gamma_t=\widehat\pi_{\min,t}\sqrt{\widehat\lambda_{\min}(\Sigma_t)+\delta}/(\widehat a_{\max,t}\varepsilon)$. Let $g_t$ be the ordinary gradient of the loss with respect to $\theta$. Update $\theta$ with $g_t/\max(\widehat\gamma_t,\gamma_0)$, clip the resulting norm, and optionally multiply by $\widehat\gamma_t/\operatorname{EMA}(\widehat\gamma_t)$ to suppress steps from poorly identifiable batches. (4) Estimate the local smoothness $\widehat L_t$ every $M$ steps with one or two Hessian-vector products and power iteration, then use $\eta_t=\eta_0/\max(\widehat L_t,L_0)$. The paper's formulas determine the conditioning diagnostic; only $\widehat L_t$ is empirically estimated. If $\widehat\gamma_t$ remains small, temporarily increase $\varepsilon$ or add covariance regularization. (5) First experiment: train a small inverse-OT model on synthetic $K=16$ transition matrices with known sparse $\theta$, comparing Adam, Adam plus clipping, and this optimizer. Measure objective versus steps and wall-clock time, parameter error, support recovery, gradient explosions, and correlation between $\widehat\gamma_t$ and subsequent loss decrease. Success means fewer failed runs and equal parameter error in fewer optimizer steps without increasing Sinkhorn iterations.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.