Spectral-gap covariance control for latent representations

Insert the regularizer on final encoder representations in a contrastive model, autoencoder, VAE encoder, or the CLS representation of a Transformer. For a minibatch \(x_1,\ldots,x_B\), compute \(h_i=f_\theta(x_i)\in\mathbb R^n\), the batch mean \(\bar h\), and covariance \(\widehat C\) using the displayed unbiased estimator. Estimate \(\|\widehat C\|_{op}\) with a differentiable three-step power iteration: initialize a unit vector \(v\), repeatedly update \(v\leftarrow\widehat C v/(\|\widehat C v\|_2+\varepsilon)\), and use \(v^\top\widehat C v\) as the eigenvalue estimate. Add \(\mathcal L_{\mathrm{torus}}\) to the task or contrastive loss. The hinge term penalizes a top eigenvalue above the target \(\tau\), corresponding to improving the lower bound \(\pi^2/(3\|C\|_{op})\); the trace term fixes average variance at \(s^2\), preventing the zero representation; the Frobenius term promotes isotropy. Warm the regularizer from zero over 1,000 steps and sweep \(\gamma,\eta,\rho\) over \(10^{-4}\) to \(10^{-1}\) relative to the primary loss. Compare with no regularizer, VICReg variance/covariance loss, and whitening. Start with CIFAR-10, a four-layer MLP or small ResNet, batch sizes 128 and 512, and standard cross-entropy. Log validation accuracy, train-loss variance, largest covariance eigenvalue, effective rank \((\operatorname{tr}C)^2/\operatorname{tr}(C^2)\), and the largest learning rate that remains stable. The expected signal is equal or higher accuracy with greater effective rank and a wider stable learning-rate range, especially at small batch sizes or under noisy labels.