# Dimension-Calibrated Ridge-OT Covariance Loss

- ID: 210
- Canonical URL: https://synthcore.org/idea/210/dimension-calibrated-ridge-ot-covariance-loss
- API JSON: https://synthcore.org/api/idea/210.json
- API Markdown: https://synthcore.org/api/idea/210.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.26693](https://arxiv.org/abs/2608.26693)
- Category: regularization
- Solves: stability, accuracy, generalization
- ML areas: loss, regularization, embedding
- Math tags: optimal-transport, random-matrix, spectral-theory, convex-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Add a Gaussian KL-UOT-inspired covariance discrepancy to a neural representation loss, using ridge-logdet terms that remain finite when minibatch covariance matrices are rank deficient. Set the unbalanced penalty to \(\tau=\kappa p\), where \(p\) is the feature dimension and \(\kappa\) is tuned over a small logarithmic grid, rather than using a dimension-independent covariance penalty. This directly tests the paper's claim that high-dimensional sample-covariance noise has a critical penalty scale proportional to dimension.

## Mathematical statement

The paper identifies an equal-marginal-penalty Gaussian KL-UOT covariance action with an exact log-determinant representation based on a nonlinear ridge product, together with a positive-semidefinite extension valid for singular matrices. The extracted text does not expose the paper's complete nonlinear-product notation, so the implementable adaptation below uses the corresponding symmetric ridge-logdet surrogate. For symmetric positive-semidefinite covariance estimates \(A,B\in\mathbb{R}^{p\times p}\), define \(A^{1/2}\) by spectral functional calculus and let \(C=(A^{1/2}BA^{1/2})^{1/2}\). The adapted loss is \(\mathcal{L}_{\tau}(A,B)=p^{-1}[\log\det(I_p+A/\tau)+\log\det(I_p+B/\tau)-2\log\det(I_p+C/\tau)]\). Here \(I_p\) is the \(p\)-dimensional identity, \(\tau>0\) is the KL/ridge penalty, and \(p\) is feature dimension. The scalar eigenvalue analogue \(\log(1+a/\tau)+\log(1+b/\tau)-2\log(1+\sqrt{ab}/\tau)\ge 0\) for \(a,b\ge0\) motivates nonnegativity and covariance alignment. The paper's random-matrix result supplies the dimension rule \(\tau_p\asymp p\): use \(\tau=\kappa p\) when minibatch size and feature dimension are comparable, with \(\kappa\) selected by validation.

## Key formulas

- $$\mathcal{L}_{\tau}(A,B)=\frac{1}{p}\left[\log\det\left(I_p+\frac{A}{\tau}\right)+\log\det\left(I_p+\frac{B}{\tau}\right)-2\log\det\left(I_p+\frac{(A^{1/2}BA^{1/2})^{1/2}}{\tau}\right)\right].$$
- $$A=\frac{1}{n-1}X_c^{\mathsf T}X_c+\varepsilon I_p,\qquad B=\frac{1}{m-1}Y_c^{\mathsf T}Y_c+\varepsilon I_p,\qquad \tau=\kappa p,$$
- $$\log(1+a/\tau)+\log(1+b/\tau)-2\log(1+\sqrt{ab}/\tau)\ge 0\qquad(a,b\ge0).$$

## Implementation notes

Integrate the loss after a selected encoder block and before the classifier head. For two augmented minibatches of the same examples, collect feature matrices X and Y with shape [batch, p]. Center each feature dimension using its minibatch mean, then compute A and B as regularized sample covariances. Use epsilon=1e-4 times the mean diagonal covariance to avoid numerical singularities. Compute the eigendecomposition A=U diag(a) U^T, form A_half=U diag(sqrt(max(a,0))) U^T, compute M=A_half B A_half, eigendecompose M, and set C=M_half. Evaluate the three log-determinants through eigenvalues, using log1p(eigenvalue/tau) instead of explicitly forming matrix inverses. The training objective is the ordinary supervised or self-supervised loss plus lambda_cov times L_tau. Pseudocode: `Xc=X-mean(X,0); Yc=Y-mean(Y,0); A=Xc.T@Xc/(n-1)+eps*I; B=Yc.T@Yc/(m-1)+eps*I; Ahalf=sym_sqrt(A); M=Ahalf@B@Ahalf; C=sym_sqrt(M); L=(sum(log1p(eig(A)/tau))+sum(log1p(eig(B)/tau))-2*sum(log1p(eig(C)/tau)))/p; loss=base+lambda_cov*L`, with `tau=kappa*p`. The mathematical elements are the ridge-logdet and symmetric PSD square-root constructions; estimate empirically the best kappa, lambda_cov, epsilon, and whether covariance matching improves validation performance. Start with a 2-layer MLP or 4-layer ConvNet on CIFAR-10 using p in {128,256,512}, batch size 64 or 128, and two standard augmentations. Compare against no covariance loss, CORAL loss, and an unscaled version with fixed tau. Success is lower representation covariance variance across minibatches, fewer exploding covariance-loss spikes, and improved test accuracy or linear-probe accuracy at equal training compute.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Implemented the dimension-calibrated ridge-logdet covariance loss with PSD square roots, covariance regularization, tau=kappa*p, and a fixed-tau comparison. The numerical check passed on rank-deficient covariances and confirmed nonnegativity/equality up to floating-point handling; proportional scaling also reduced loss variance and magnitude at larger dimensions. However, the learning experiment produced identical 99.22% test accuracy for all methods, and calibrated regularization did not improve held-out covariance discrepancy, so no practical task win was observed.

### Mechanism check

- Verdict: Implemented the dimension-calibrated ridge-logdet covariance loss with PSD square roots, covariance regularization, tau=kappa*p, and a fixed-tau comparison. The numerical check passed on rank-deficient covariances and confirmed nonnegativity/equality up to floating-point handling; proportional scaling also reduced loss variance and magnitude at larger dimensions. However, the learning experiment produced identical 99.22% test accuracy for all methods, and calibrated regularization did not improve held-out covariance discrepancy, so no practical task win was observed.
- Confidence: 9/10
- Limitations: Only a small synthetic two-class MLP experiment was tested, with one seed and short training. CIFAR-10, CORAL, broader kappa/lambda sweeps, multiple seeds, FLOP-normalized comparisons, and representation quality or downstream linear-probe evaluation were not tested.

## Artifacts

- [experiment.py](https://synthcore.org/code/33/experiment.py)
- [report.md](https://synthcore.org/code/33/report.md)
- [Download all files as ZIP](https://synthcore.org/download/33)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.