# Centered-Geometry Projection Loss

- ID: 3043
- Canonical URL: https://synthcore.org/idea/3043/centered-geometry-projection-loss
- API JSON: https://synthcore.org/api/idea/3043.json
- API Markdown: https://synthcore.org/api/idea/3043.md
- Verification status: failed_benchmark
- Source: [arXiv:2609.02155](https://arxiv.org/abs/2609.02155)
- Category: regularization
- Solves: accuracy, generalization, memory
- ML areas: embedding, loss, regularization, inference-speedup
- Math tags: probability, statistics, linear-algebra, information-theory
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Train a low-dimensional projection of embeddings against centered pairwise geometry instead of only using raw-distance preservation or a JL-style guarantee. The loss removes the population or minibatch distance baseline before comparing distances, forcing the bottleneck to retain the fluctuations that carry ranking and task information.

## Mathematical statement

For two independent isotropic Gaussian points x,y distributed as N(0,\sigma^{2}I_d), let D=\|x-y\|^{2} and let the projected representation be z=\sqrt{d/m}\,Px, where P has orthonormal rows, so the rescaled projected distance is D_m=(d/m)\|P(x-y)\|^{2}. The paper distinguishes the baseline E[D]=2d\sigma^{2} from the centered fluctuation D-E[D]. The optimal squared-error decoder of any distance feature f(D) from a sketch S is E[f(D)\mid S]. For a rank-m sketch, the squared singular values of this conditional-expectation operator imply that at most an m/d fraction of the variance of any feature of one squared distance is retained. The paper also gives expected Kendall ranking correlation \rho_K=(2/\pi)\sqrt{m/d}(1+o(1)) when m,d    o\infty and m/d    o0. We exploit these facts by explicitly matching centered distances: for pair (i,j), \widetilde D_{ij}=D_{ij}-b and \widetilde D^{(m)}_{ij}=D^{(m)}_{ij}-b_m, where b and b_m are stop-gradient batch means or running estimates.

## Key formulas

- $$D=\|x-y\|^{2},\qquad \mathbb{E}[D]=2d\sigma^{2},\qquad D_m=\frac{d}{m}\|P(x-y)\|^{2}.$$
- $$\widetilde D_{ij}=D_{ij}-b,\qquad \widetilde D^{(m)}_{ij}=D^{(m)}_{ij}-b_m.$$
- $$\operatorname{Var}\!\left(\mathbb{E}[f(D)\mid S]\right)\leq \frac{m}{d}\operatorname{Var}(f(D)).$$
- $$\rho_K=\frac{2}{\pi}\sqrt{\frac{m}{d}}\,(1+o(1)).$$

## Implementation notes

Integrate this at an embedding projection bottleneck: an encoder produces h_i in R^d, then a learned matrix W produces z_i=Wh_i in R^m, with m much smaller than d. During training, sample a minibatch of B embeddings and construct K random or semi-hard pairs. Compute raw distances D_ij=||h_i-h_j||^2 and projected distances Dm_ij=||z_i-z_j||^2; rescale Dm by d/m if W is initialized as an approximately isotropic projection. Estimate baselines b and bm as exponential moving averages of the batch means, with stop-gradient on both baselines. Add L_center=(1/K) sum_ij (standardize(D_ij-b)-standardize(Dm_ij-bm))^2, or use a ranking variant based on pairwise signs. A compact update is: h=encoder(x); z=W h; D=pdist2(h); Dm=pdist2(z)*d/m; b=EMA(mean(D)); bm=EMA(mean(Dm)); loss=task_loss+lambda*MSE((D-b)/std(D),(Dm-bm)/std(Dm)); update encoder and W. The paper supplies the m/d variance ceiling and sqrt(m/d) ranking prediction; batch baselines, normalization, and lambda are empirical. First test on CIFAR-100 or a sentence-embedding dataset with d=768 and m in {64,128,256}. Compare a plain learned projection, a fixed JL random projection, and this loss on Recall@1, Kendall correlation of pairwise rankings, downstream accuracy, and embedding memory at equal m. Success is higher Recall@1 or ranking correlation at the same m, with less than 10% training overhead and no degradation in the original task loss.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a self-contained centered-geometry projection-loss experiment with Gaussian ranking verification and a matched learned bottleneck comparison. The Gaussian Kendall correlation followed the claimed (2/pi)sqrt(m/d) scaling, and centered training improved Kendall ranking from 0.613 to 0.679 and normalized centered distance MSE from 0.351 to 0.237. However, nearest-neighbor recall decreased from 0.238 to 0.179 and raw distance preservation became much worse, so the benefit is specifically for centered geometry/ranking rather than universal embedding quality.

### Mechanism check

- Verdict: Built a self-contained centered-geometry projection-loss experiment with Gaussian ranking verification and a matched learned bottleneck comparison. The Gaussian Kendall correlation followed the claimed (2/pi)sqrt(m/d) scaling, and centered training improved Kendall ranking from 0.613 to 0.679 and normalized centered distance MSE from 0.351 to 0.237. However, nearest-neighbor recall decreased from 0.238 to 0.179 and raw distance preservation became much worse, so the benefit is specifically for centered geometry/ranking rather than universal embedding quality.
- Confidence: 8/10
- Limitations: This is a small synthetic fixed-embedding experiment, not CIFAR-100 or a real sentence-embedding benchmark. It compares current-batch centering rather than a separately tuned EMA baseline, uses one bottleneck size, and does not measure wall-clock overhead or equal-FLOP training.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.9445
- Paired wins: 4/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/1223/bench_report.json)
- [experiment.py](https://synthcore.org/code/1223/experiment.py)
- [report.md](https://synthcore.org/code/1223/report.md)
- [report_bench_2026-09-03T124900.md](https://synthcore.org/code/1223/report_bench_2026-09-03T124900.md)
- [results.json](https://synthcore.org/code/1223/results.json)
- [stage2_bench.py](https://synthcore.org/code/1223/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1223)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.