Matrix-perspective feature divergence

Integrate the method into a teacher–student distillation or representation-matching pipeline at the feature covariance tensor, rather than directly at logits. For a minibatch of size m, let H_T,H_S in R^(m x d) be teacher and student features after projection to a common dimension d. Center both feature matrices identically and form A=H_T H_T^T/m and B=H_S H_S^T/m; add epsilon I to both matrices, with epsilon between 1e-5 and 1e-3. The implementation is: HT=HT-HT.mean(0); HS=HS-HS.mean(0); [email protected]/m+eps*I; [email protected]/m+eps*I; L=cholesky(B); C=solve(L,A); C=solve(L.T,C.T).T; C=(C+C.T)/2; lam,Q=eigh(C); lam=clamp(lam,min=1e-6); F=lam*log(lam); D=L.T@Q@diag(F)@Q.T@L; loss_mp=trace(D)/m; total=task_loss+lambda_mp*loss_mp. Use a symmetric eigendecomposition and gradient clipping if necessary. The paper supplies the matrix perspective and the operator-convex choice f(t)=t log t; covariance formation, ridge regularization, projection dimension, and trace reduction are engineering choices. The first cheap experiment is CIFAR-10 with a frozen wider ResNet teacher and a ResNet-18 student, comparing logit KL, covariance Frobenius matching, and this loss at equal training FLOPs. Measure accuracy, calibration error, gradient-norm variance, and robustness to feature rotations. Success means higher accuracy or calibration at equal distillation loss, with fewer unstable runs and reduced degradation under small batch changes.