# Conditioned PSD sensing bottleneck

- ID: 58
- Canonical URL: https://synthcore.org/idea/58/conditioned-psd-sensing-bottleneck
- API JSON: https://synthcore.org/api/idea/58.json
- API Markdown: https://synthcore.org/api/idea/58.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.22418](https://arxiv.org/abs/2608.22418)
- Category: regularization
- Solves: stability, accuracy, memory
- ML areas: embedding, architecture, regularization
- Math tags: linear-algebra, metric-geometry, optimization
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Represent an intermediate feature as a low-rank PSD matrix and compress it using nonnegative measurements \(\langle A_i,X\rangle\), while penalizing the empirical ratio between maximum and minimum measurement distortion over low-rank feature pairs. This directly discourages collapsed directions and excessively amplified directions in a covariance or Gram-feature bottleneck.

## Mathematical statement

The paper defines the PSD measurement map \(\Phi_{\mathcal A}(X)=(\langle A_1,X\rangle,\ldots,\langle A_m,X\rangle)\), where \(A_i\succeq0\), \(X\succeq0\), and \(\langle A,X\rangle=\operatorname{Tr}(A^*X)\). For rank at most \(r\), the lower and upper \(\ell_1\)-to-Frobenius Lipschitz constants are \(L_{\mathcal A}(r)=\inf_{X\ne Y}\|\Phi_{\mathcal A}(X)-\Phi_{\mathcal A}(Y)\|_1/\|X-Y\|_F\) and \(U_{\mathcal A}(r)=\sup_{X\ne Y}\|\Phi_{\mathcal A}(X)-\Phi_{\mathcal A}(Y)\|_1/\|X-Y\|_F\), with both \(X,Y\) PSD and rank at most \(r\). The condition number is \(\beta_{\mathcal A}(r)=U_{\mathcal A}(r)/L_{\mathcal A}(r)\). The neural adaptation uses minibatch estimates of these quantities; a large empirical \(L\) prevents indistinguishable representations, while a small \(U\) prevents unstable amplification.

## Key formulas

- $$\Phi_{\mathcal{A}}(X):=(\langle A_{1},X\rangle,\dots,\langle A_{m},X\rangle),\qquad \langle A,X\rangle:=\operatorname{Tr}(A^{*}X).$$
- $$L_{\mathcal A}(r):=\inf_{\substack{X,Y\succeq0,\;\operatorname{rank}(X),\operatorname{rank}(Y)\le r\\X\ne Y}}\frac{\|\Phi_{\mathcal A}(X)-\Phi_{\mathcal A}(Y)\|_1}{\|X-Y\|_F},\qquad U_{\mathcal A}(r):=\sup_{\substack{X,Y\succeq0,\;\operatorname{rank}(X),\operatorname{rank}(Y)\le r\\X\ne Y}}\frac{\|\Phi_{\mathcal A}(X)-\Phi_{\mathcal A}(Y)\|_1}{\|X-Y\|_F}.$$
- $$\beta_{\mathcal A}(r):=\frac{U_{\mathcal A}(r)}{L_{\mathcal A}(r)}.$$
- $$\mathcal L_{\mathrm{cond}}=\log(\widehat U+\varepsilon)-\log(\widehat L+\varepsilon),\quad q_j=\frac{\|\Phi_{\mathcal A}(X_j)-\Phi_{\mathcal A}(Y_j)\|_1}{\|X_j-Y_j\|_F+\varepsilon},\quad \widehat U=\operatorname{quantile}_{0.95}(\{q_j\}),\ \widehat L=\operatorname{quantile}_{0.05}(\{q_j\}).$$

## Implementation notes

Use this as a replacement for a generic projection bottleneck in a model that naturally produces covariance or Gram features. At an intermediate layer, let \(H\in\mathbb R^{d\times s}\) be a token-feature matrix, form \(X=HH^\top/(s+\varepsilon)\in\mathbb S_d^+\), and parameterize each sensing matrix as \(A_i=a_i a_i^\top\), so the bottleneck output is \(z_i=\langle A_i,X\rangle=\|a_i^\top H\|_2^2/(s+\varepsilon)\). During training, retain two augmented or independently sampled views of each example, producing \(X_j,Y_j\), and compute \(q_j\) using the formula above. Use detached 5th and 95th percentile estimates for \(\widehat L\) and \(\widehat U\), add \(\lambda\mathcal L_{\mathrm{cond}}\) to the task loss, and stop gradients through quantile selection if optimization is noisy. Pseudocode: `H1,H2=model_features(view1,view2); X=H1@H1.T/(s+eps); Y=H2@H2.T/(s+eps); zX=(A@H1).square().sum(1)/s; zY=(A@H2).square().sum(1)/s; q=abs(zX-zY).sum()/(norm(X-Y,'fro')+eps); Lhat=quantile(q,.05); Uhat=quantile(q,.95); loss=task_loss+lambda*(log(Uhat+eps)-log(Lhat+eps))`. First test on CIFAR-10 with a small ResNet or ViT, using \(d=64,m=128\), against a learned dense projection and an unregularized PSD bottleneck. Measure validation accuracy, perturbation consistency, empirical condition ratio, and accuracy after reducing \(m\). Success means a larger lower distortion and improved robustness at equal bottleneck width.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a rank-one PSD sensing bottleneck toy verifier with quantile condition loss and optimized sensing vectors. All three mechanism predictions manifested: beta was invariant under common scaling while L and U scaled by 3.7^2, beta decreased from 3.281 to 1.237 as m increased from 8 to 256, and beta increased from 1.321 to 9.408 as anisotropy gamma increased from 1 to 16. At m=32, conditioning reduced test beta from 1.799 to 1.367 and increased L from 2.608 to 3.035, so the proposed effect is real in this controlled setting.

### Mechanism check

- Verdict: Built a rank-one PSD sensing bottleneck toy verifier with quantile condition loss and optimized sensing vectors. All three mechanism predictions manifested: beta was invariant under common scaling while L and U scaled by 3.7^2, beta decreased from 3.281 to 1.237 as m increased from 8 to 256, and beta increased from 1.321 to 9.408 as anisotropy gamma increased from 1 to 16. At m=32, conditioning reduced test beta from 1.799 to 1.367 and increased L from 2.608 to 3.035, so the proposed effect is real in this controlled setting.
- Confidence: 9/10
- Limitations: Only synthetic rank-one PSD feature pairs were tested; no CIFAR-10 model, dense projection baseline, task accuracy, perturbation robustness, FLOPs, memory, or multi-seed evaluation was run. The width and anisotropy sweeps use empirical quantiles rather than exact global Lipschitz constants.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/698/bench_report.json)
- [experiment.py](https://synthcore.org/code/698/experiment.py)
- [report.md](https://synthcore.org/code/698/report.md)
- [report_bench_2026-08-31T222212.md](https://synthcore.org/code/698/report_bench_2026-08-31T222212.md)
- [results.json](https://synthcore.org/code/698/results.json)
- [results_final.json](https://synthcore.org/code/698/results_final.json)
- [stage2_psd_bench.py](https://synthcore.org/code/698/stage2_psd_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/698)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.