✗ Failed on benchmark 2026

Centered-Geometry Projection Loss

Implementation & benchmark of arXiv:2609.02155 — Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models

Usefulness7/10

Difficulty4/10

Novelty6/10

Regularization Accuracy Generalization Memory Embedding Loss Regularization Inference speedup Probability Statistics Linear algebra Information theory

Source paper: Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models arXiv:2609.02155 ⓘ · analyzed Sep 3, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Train a low-dimensional projection of embeddings against centered pairwise geometry instead of only using raw-distance preservation or a JL-style guarantee. The loss removes the population or minibatch distance baseline before comparing distances, forcing the bottleneck to retain the fluctuations that carry ranking and task information.

Formulas

$$D=\|x-y\|^{2},\qquad \mathbb{E}[D]=2d\sigma^{2},\qquad D_m=\frac{d}{m}\|P(x-y)\|^{2}.$$

$$\widetilde D_{ij}=D_{ij}-b,\qquad \widetilde D^{(m)}_{ij}=D^{(m)}_{ij}-b_m.$$

$$\operatorname{Var}\!\left(\mathbb{E}[f(D)\mid S]\right)\leq \frac{m}{d}\operatorname{Var}(f(D)).$$

$$\rho_K=\frac{2}{\pi}\sqrt{\frac{m}{d}}\,(1+o(1)).$$

Mathematical statement

For two independent isotropic Gaussian points x,y distributed as N(0,\sigma^{2}I_d), let D=\|x-y\|^{2} and let the projected representation be z=\sqrt{d/m}\,Px, where P has orthonormal rows, so the rescaled projected distance is D_m=(d/m)\|P(x-y)\|^{2}. The paper distinguishes the baseline E[D]=2d\sigma^{2} from the centered fluctuation D-E[D]. The optimal squared-error decoder of any distance feature f(D) from a sketch S is E[f(D)\mid S]. For a rank-m sketch, the squared singular values of this conditional-expectation operator imply that at most an m/d fraction of the variance of any feature of one squared distance is retained. The paper also gives expected Kendall ranking correlation \rho_K=(2/\pi)\sqrt{m/d}(1+o(1)) when m,d o\infty and m/d o0. We exploit these facts by explicitly matching centered distances: for pair (i,j), \widetilde D_{ij}=D_{ij}-b and \widetilde D^{(m)}_{ij}=D^{(m)}_{ij}-b_m, where b and b_m are stop-gradient batch means or running estimates.

Implementation notes

Integrate this at an embedding projection bottleneck: an encoder produces h_i in R^d, then a learned matrix W produces z_i=Wh_i in R^m, with m much smaller than d. During training, sample a minibatch of B embeddings and construct K random or semi-hard pairs. Compute raw distances D_ij=||h_i-h_j||^2 and projected distances Dm_ij=||z_i-z_j||^2; rescale Dm by d/m if W is initialized as an approximately isotropic projection. Estimate baselines b and bm as exponential moving averages of the batch means, with stop-gradient on both baselines. Add L_center=(1/K) sum_ij (standardize(D_ij-b)-standardize(Dm_ij-bm))^2, or use a ranking variant based on pairwise signs. A compact update is: h=encoder(x); z=W h; D=pdist2(h); Dm=pdist2(z)*d/m; b=EMA(mean(D)); bm=EMA(mean(Dm)); loss=task_loss+lambda*MSE((D-b)/std(D),(Dm-bm)/std(Dm)); update encoder and W. The paper supplies the m/d variance ceiling and sqrt(m/d) ranking prediction; batch baselines, normalization, and lambda are empirical. First test on CIFAR-100 or a sentence-embedding dataset with d=768 and m in {64,128,256}. Compare a plain learned projection, a fixed JL random projection, and this loss on Recall@1, Kendall correlation of pairwise rankings, downstream accuracy, and embedding memory at equal m. Success is higher Recall@1 or ranking correlation at the same m, with less than 10% training overhead and no degradation in the original task loss.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a self-contained centered-geometry projection-loss experiment with Gaussian ranking verification and a matched learned bottleneck comparison. The Gaussian Kendall correlation followed the claimed (2/pi)sqrt(m/d) scaling, and centered training improved Kendall ranking from 0.613 to 0.679 and normalized centered distance MSE from 0.351 to 0.237. However, nearest-neighbor recall decreased from 0.238 to 0.179 and raw distance preservation became much worse, so the benefit is specifically for centered geometry/ranking rather than universal embedding quality.

Agent confidence: 8/10

Baseline: Kendall tau 0.6129; centered normalized MSE 0.3514; raw normalized MSE 0.7889; Recall@1 0.2375
Gaussian check: Observed Kendall tau for m={2,4,8,16,32}: {0.097,0.148,0.223,0.320,0.495}; predictions: {0.113,0.159,0.225,0.318,0.450}
Idea: Kendall tau 0.6795; centered normalized MSE 0.2373; raw normalized MSE 1041.7; Recall@1 0.1792

Limitations: This is a small synthetic fixed-embedding experiment, not CIFAR-100 or a real sentence-embedding benchmark. It compares current-batch centering rather than a separately tuned EMA baseline, uses one bottleneck size, and does not measure wall-clock overhead or equal-FLOP training.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.02155) Sep 3, 2026

Ahead of the baseline (−0.3%) but not statistically significant (p=0.9445, wins 4 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp
Paired seeds: 8
Baseline mean: 7.655
Idea mean: 7.6314
Effect (Δ): -0.0237 (−0.3%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.9445 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±1.5%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the registered tabular track was tested; vision, sequence, and dynamics were not tested. The compact run used 400 training samples, 200 test samples, and 24 epochs. The mechanism signature was measured on the trained idea model, but the Gaussian rank-sketch formula does not directly predict behavior for its learned nonlinear encoder.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 3, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Centered-Geometry Projection Loss in a shared tabular MLP with a 32-dimensional hidden representation and 8-dimensional learned bottleneck. The full 8-seed comparison yielded a tiny improvement in test MSE, 7.631358 versus 7.655009, but paired delta was -0.023651 with permutation p=0.9445, giving no measurable effect. The trained-model signature observed centered-distance correlation 0.999976 but variance ratio 98.0545 versus the reported <=1 ceiling, so the mechanism prediction was not confirmed.

Baseline: Registered tabular/mlp track; tuned baseline lr=0.01; full mean test MSE 7.6550092101, std 0.7549298959.
Idea: Selected lr=0.01, lambda=0.03; full mean test MSE 7.6313580275, std 0.8269721814; paired delta -0.0236511827, p=0.9445, idea wins 4/8.
Mechanism signature: map[confirmed:false observed_centered_distance_correlation:0.9999756813 observed_centered_variance_ratio:98.0545272827 predicted_variance_ceiling:1]

Benchmark result