# Orthogonal supervised coordinate bottleneck

- ID: 246
- Canonical URL: https://synthcore.org/idea/246/orthogonal-supervised-coordinate-bottleneck
- API JSON: https://synthcore.org/api/idea/246.json
- API Markdown: https://synthcore.org/api/idea/246.md
- Verification status: unverified
- Source: [arXiv:2607.03692](https://arxiv.org/abs/2607.03692)
- Category: architecture
- Solves: generalization, memory, accuracy
- ML areas: embedding, mlp, loss, memory
- Math tags: functional-analysis, linear-algebra, spectral-theory, regularization
- Ratings: usefulness 6/10; difficulty 3/10; novelty 5/10

## Idea description

Make a compact hidden representation explicitly decorrelated under the empirical data distribution while retaining a supervised linear readout. This creates a spectral-style bottleneck whose coordinates cannot redundantly encode the same feature, potentially improving small embeddings and making downstream linear decoding more effective.

## Mathematical statement

Use the paper's population inner product \(\langle f_1,f_2\rangle_0=\int f_1(x)f_2(x)p(x)dx\) as the target notion of coordinate orthogonality. For a minibatch representation matrix \(Z\in\mathbb{R}^{B\times k}\), with row \(z_n=f_\theta(x_n)\), center the features as \(\widetilde Z=Z-\mathbf{1}\bar z^{\mathsf T}\), where \(\bar z=B^{-1}\sum_n z_n\), and form the empirical Gram matrix \(C=B^{-1}\widetilde Z^{\mathsf T}\widetilde Z\). Penalize deviation from identity with \(L_{gram}=\|C-I_k\|_F^2\), a batch approximation to \(\langle z_i,z_j\rangle_0=\delta_{ij}\). Couple this with a supervised linear readout \(g(x)=Wz(x)+b\), where \(W\in\mathbb{R}^{c\times k}\) maps the \(k\)-dimensional representation to \(c\) classes, and optionally with the metric-weighted Dirichlet energy \(L_{dir}=B^{-1}\sum_n\|J_z(x_n)A(x_n)^T\|_F^2\). The identity Gram target fixes coordinate scale and discourages collapse, while the linear readout ensures that the coordinates remain predictive rather than merely diverse.

## Key formulas

- $$\langle f_{1},f_{2}\rangle_{0}=\int f_{1}(\mathbf{x})f_{2}(\mathbf{x})p(\mathbf{x})d\mathbf{x}$$
- $$C=\frac{1}{B}\widetilde Z^{\mathsf T}\widetilde Z,\qquad \widetilde Z=Z-\mathbf{1}\bar z^{\mathsf T},\qquad L_{\mathrm{gram}}=\|C-I_k\|_{F}^{2}$$
- $$L(\theta,W)=L_{\mathrm{task}}(WZ)+\lambda_{\mathrm{gram}}\left\|\frac{1}{B}\widetilde Z^{\mathsf T}\widetilde Z-I_k\right\|_{F}^{2}+\lambda_{\mathrm{dir}}\frac{1}{B}\sum_{n=1}^{B}\left\|J_z(x_n)A(x_n)^{\mathsf T}\right\|_{F}^{2}$$
- $$\min_{\theta,W}\;L_{\mathrm{task}}(Wf_\theta(x))+\lambda_{\mathrm{gram}}L_{\mathrm{gram}}+\lambda_{\mathrm{dir}}L_{\mathrm{dir}},\qquad W\in\mathbb{R}^{c\times k}$$

## Implementation notes

Insert a projection head \(z=f_\theta(x)\in\mathbb{R}^k\) before an existing classifier or encoder output, choosing \(k\) much smaller than the original hidden width. On every training step, compute \(Z=f_\theta(X)\), the batch mean \(\bar z=B^{-1}\sum_nz_n\), centered features \(\widetilde Z=Z-\mathbf{1}\bar z^T\), and \(C=B^{-1}\widetilde Z^T\widetilde Z\). Add \(\lambda_{gram}\|C-I_k\|_F^2\) to the task loss. Normalize the representation with LayerNorm before the projection, or otherwise control feature scale, because unconstrained scaling can interact with the identity target. Keep the task head explicitly linear, \(\mathrm{logits}=WZ+b\), matching the supervised readout in the construction. The core pseudocode is: `Z=proj(encoder(X)); Zc=Z-Z.mean(0); C=Zc.T@Zc/B; loss=CE(W@Z+b,y)+lam_gram*((C-I)**2).sum(); loss.backward(); optimizer.step()`. The identity Gram matrix is the mathematical target; minibatch covariance is its empirical estimator. Track an exponential moving average of \(C\) only for diagnostics, unless a large-batch variant is desired. First test CIFAR-10 with a small CNN or ViT and a 64-dimensional projection, comparing cross-entropy alone, ordinary covariance decorrelation, Barlow-Twins-style off-diagonal loss, and the full identity-Gram penalty. Evaluate accuracy versus projection dimension, linear-probe accuracy, covariance off-diagonal energy, and representation memory. Success means matching baseline accuracy with a 2x--4x smaller representation and lower coordinate redundancy without materially slower optimization.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.