Spherical anti-additive-collision embeddings
Implementation & benchmark of arXiv:2608.14467 — Near diagonal additive energy bound for points on algebraic surfaces
Source paper: Near diagonal additive energy bound for points on algebraic surfaces arXiv:2608.14467 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Constrain an embedding table to the unit sphere and penalize repeated or nearly repeated pair sums. This discourages additive quadruples a+b approximately equal to c+d, reducing unwanted linear structure and making distinct tokens less interchangeable under downstream composition. The theorem provides a geometric target: on a sphere, the affine-line concentration factor is bounded by two, so exact additive energy should scale close to n squared rather than the much larger values produced by collapsed or arithmetic-progression-like embeddings.
Formulas
Mathematical statement
For finite sets X,Y in R^n, the paper defines d_{X-Y}(t)=#\{(x,y) in X times Y: x-y=t\} and E(X,Y)=sum_t d_{X-Y}(t)^2. It uses the identity E(X,Y)=sum_t d_X(t)d_Y(t), where d_X=d_{X-X}. For a polynomial surface Z(F), it defines Lambda_F(X)=max\{1,sup_{ell an affine line in Z(F)} #(X intersect ell)\}. Theorem 1.1 states that for irreducible real F with degree at least 2, every finite X subset Z(F) satisfies E(X) <= C_epsilon Lambda_F(X)(#X)^(2+epsilon). Here E(X)=#\{(a,b,c,d) in X^4: a+b=c+d\}. For the adapted model, use F(z)=||z||_2^2-1, so Z(F) is the unit sphere; every affine line intersects the sphere in at most two points, giving Lambda_F(X)<=2. Replace exact equality by the differentiable kernel K_tau(u)=exp(-||u||_2^2/(2 tau^2)) and minimize the excess soft energy over nontrivial quadruples.
Implementation notes
(1) Integration point: apply this to an embedding or codebook matrix U in R^(n times d), such as a vocabulary embedding table or VQ-VAE codebook. Normalize each row to z_i=u_i/(||u_i||_2+10^(-8)) before lookup, so all code vectors lie on the unit sphere. Add lambda_AE times the regularizer to the ordinary task loss, using only a sampled subset of rows or refreshing the subset periodically. (2) Pseudocode: sample m distinct indices I; draw B quadruples (i,j,k,l) from I^4; compute r=z_i+z_j-z_k-z_l and q=exp(-||r||_2^2/(2 tau^2)); exclude trivial permutations where (i,j) equals (k,l) or (l,k); average q to obtain E_hat_nontriv; optimize L_total=L_task+lambda_AE times max(0,E_hat_nontriv-rho)^2. Anneal tau from 0.2 to 0.03 in normalized units. (3) The exact mathematical object is E(X)=sum_t d_X(t)^2; training uses the Gaussian relaxation because exact equalities have measure zero for floating-point embeddings. Estimate the relaxation with random quadruples, and independently validate it with 100,000 held-out quadruples. A cheaper approximation quantizes every pair sum z_i+z_j into grid cells of width tau, counts occupancies h(c), and uses sum_c h(c)^2 as a discretized additive-energy estimate. (4) First experiment: train a 2-layer Transformer language model on WikiText-2 with a 10,000-token vocabulary. Compare unconstrained embeddings, spherical normalization alone, and spherical normalization plus the regularizer at lambda_AE values 10^(-4), 10^(-3), and 10^(-2), with matched parameter count and compute. Record validation perplexity, nontrivial soft energy, nearest-neighbor overlap, and token-context linear-probe accuracy. Success means lower collision energy with equal or better perplexity; failure means no energy reduction or degraded perplexity and semantic neighborhoods.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.