Orthogonal supervised coordinate bottleneck

Insert a projection head \(z=f_\theta(x)\in\mathbb{R}^k\) before an existing classifier or encoder output, choosing \(k\) much smaller than the original hidden width. On every training step, compute \(Z=f_\theta(X)\), the batch mean \(\bar z=B^{-1}\sum_nz_n\), centered features \(\widetilde Z=Z-\mathbf{1}\bar z^T\), and \(C=B^{-1}\widetilde Z^T\widetilde Z\). Add \(\lambda_{gram}\|C-I_k\|_F^2\) to the task loss. Normalize the representation with LayerNorm before the projection, or otherwise control feature scale, because unconstrained scaling can interact with the identity target. Keep the task head explicitly linear, \(\mathrm{logits}=WZ+b\), matching the supervised readout in the construction. The core pseudocode is: Z=proj(encoder(X)); Zc=Z-Z.mean(0); C=Zc.T@Zc/B; loss=CE(W@Z+b,y)+lam_gram*((C-I)**2).sum(); loss.backward(); optimizer.step(). The identity Gram matrix is the mathematical target; minibatch covariance is its empirical estimator. Track an exponential moving average of \(C\) only for diagnostics, unless a large-batch variant is desired. First test CIFAR-10 with a small CNN or ViT and a 64-dimensional projection, comparing cross-entropy alone, ordinary covariance decorrelation, Barlow-Twins-style off-diagonal loss, and the full identity-Gram penalty. Evaluate accuracy versus projection dimension, linear-probe accuracy, covariance off-diagonal energy, and representation memory. Success means matching baseline accuracy with a 2x--4x smaller representation and lower coordinate redundancy without materially slower optimization.