Spectrally Projected SPD Optimizer

(1) Integration point: modify the optimizer update for a small symmetric matrix parameter \(S\), rather than for ordinary unconstrained neural-network weights. Use cases include a learned Mahalanobis metric, a VAE covariance output, a graph message-passing kernel, or a quantum density-matrix head. Store the full matrix or upper triangle, symmetrize each gradient, and project after every optimizer proposal. (2) Pseudocode: compute loss \(L\), obtain \(G=\nabla_S L\), set \(G\leftarrow(G+G^\top)/2\), apply SGD or AdamW to obtain proposal \(X\), symmetrize \(X\leftarrow(X+X^\top)/2\), compute \(X=U\operatorname{diag}(\lambda)U^\top\), and assign \(S\leftarrow U\operatorname{diag}(\operatorname{clip}(\lambda,\alpha,\beta))U^\top\). Apply weight decay before projection and do not decay the identity floor. (3) Compute the projection with a symmetric eigensolver. Select \(\alpha\) as a numerical floor such as \(10^{-4}\), and set \(\beta=\alpha\times10^3\) or tune the target condition number. Log the fraction of clipped eigenvalues, minimum and maximum eigenvalues, and the empirical BW displacement before and after projection. The theorem supplies the non-expansiveness result; convergence and generalization improvements must be tested. (4) First experiment: train a two-layer metric-learning MLP on MNIST or Fashion-MNIST with a 32-by-32 SPD Mahalanobis matrix, comparing Cholesky parameterization, unconstrained symmetric-plus-jitter updates, and projected updates under SGD and AdamW. Track training loss, test k-nearest-neighbor accuracy under the learned metric, gradient norms, eigenvalue ranges, failed runs, and seed variance. Success is zero SPD failures, bounded condition number, and equal-or-better test accuracy at matched optimizer steps; a stronger result is faster loss descent or reduced seed variance at the same matrix-eigendecomposition overhead.