# Convex-lifted additive codebook

- ID: 2178
- Canonical URL: https://synthcore.org/idea/2178/convex-lifted-additive-codebook
- API JSON: https://synthcore.org/api/idea/2178.json
- API Markdown: https://synthcore.org/api/idea/2178.md
- Verification status: unverified
- Source: [arXiv:2608.12316](https://arxiv.org/abs/2608.12316)
- Category: architecture
- Solves: accuracy, generalization
- ML areas: embedding, retrieval, transformer
- Math tags: combinatorics, geometry, convex-analysis
- Ratings: usefulness 5/10; difficulty 3/10; novelty 7/10

## Idea description

Encode each scalar or discrete code t by the strictly convex lift γ(t)=(t,t²), optionally followed by a learned linear projection and normalization. Because three-code sums on this curve have only near-minimal additive energy, the representation should produce fewer collisions when a model composes three tokens, codes, or retrieved items by addition.

## Mathematical statement

The paper defines the k-fold additive energy of a finite set Y⊂R^n as J_k(Y):=#{(y_1,…,y_{2k})∈Y^{2k}: Σ_{i=1}^k y_i=Σ_{i=k+1}^{2k} y_i}. A function f:R→R is strictly convex when f(λx+(1−λ)y)<λf(x)+(1−λ)f(y) for distinct x,y and λ∈(0,1), and the associated curve is γ(t)=(t,f(t)). Theorem 1.1 gives J_3(γ(X))≪_ε|X|^{3+ε} for every finite X⊂R and ε>0, where |X| is the codebook size and the implicit constant depends on ε. Therefore the number of collisions between sums of three lifted codewords is close to the diagonal lower-bound scale |X|^3. Choosing f(t)=t² satisfies strict convexity. For this choice, equality of triple sums is equivalent to simultaneous equality of the first and second power sums: Σ_{i=1}^3x_i=Σ_{i=4}^6x_i and Σ_{i=1}^3x_i²=Σ_{i=4}^6x_i².

## Key formulas

- $$J_k(Y):=\#\left\{(y_1,\ldots,y_{2k})\in Y^{2k}:\sum_{i=1}^{k}y_i=\sum_{i=k+1}^{2k}y_i\right\}.$$
- $$\gamma(t)=(t,f(t)),\qquad f(\lambda x+(1-\lambda)y)<\lambda f(x)+(1-\lambda)f(y).$$
- $$J_3(\gamma(X))=\#\left\{(x_1,\ldots,x_6)\in X^6:\sum_{i=1}^{3}\gamma(x_i)=\sum_{i=4}^{6}\gamma(x_i)\right\}\ll_{\epsilon}|X|^{3+\epsilon}.$$
- $$\gamma_{\mathrm{ML}}(t)=\bigl(t,t^2\bigr),\qquad \sum_{i=1}^3\gamma_{\mathrm{ML}}(x_i)=\sum_{i=4}^6\gamma_{\mathrm{ML}}(x_i)\iff\begin{cases}\sum_{i=1}^3x_i=\sum_{i=4}^6x_i,\\\sum_{i=1}^3x_i^2=\sum_{i=4}^6x_i^2.\end{cases}$$

## Implementation notes

(1) Integration point: use the construction as the representation layer for a finite scalar codebook, tokenizer IDs, quantized latent values, or compositional retrieval keys. For N ordered codes, set t_j=2j/(N−1)−1 and replace the ordinary scalar representation t_j with z_j=(t_j,t_j²). Concatenate z_j to the standard embedding, or apply a learned matrix P∈R^{d×2} initialized with orthogonal columns. Keep P fixed in one ablation and trainable in another. The energy guarantee applies directly before arbitrary nonlinear transformations, so collision measurements should be made on z_j or on an invertible linear transform.
(2) Pseudocode: create t=linspace(−1,1,N), z=stack([t,t*t],axis=−1). For each sampled triple (j1,j2,j3), compute s=z[j1]+z[j2]+z[j3]. Sort exact rational or quantized sums and count duplicate pairs to estimate J_3. For continuous floating-point experiments, normalize s and count pairs satisfying ||s_a−s_b||₂<τ. Feed z, or Pz, into the downstream model and train with the ordinary task loss. Optionally add λ exp(−||s_a−s_b||₂²/τ²) over distinct sampled triples as a differentiable near-collision penalty when t or P is trainable.
(3) The theorem supplies the strict-convexity construction and the near-cubic exact-energy target. The implementation must empirically estimate approximate energy because finite precision, projection, normalization, and noise are not covered by the theorem.
(4) First experiment: use N=256 codes in a synthetic task whose labels depend on unordered triples. Compare a learned scalar embedding, random two-dimensional embeddings, and fixed (t,t²) embeddings at matched dimension and parameter count. Measure the number of unique triple sums, the number of pairs within thresholds τ∈{10^−3,10^−2,10^−1}, noisy top-1 triple decoding accuracy, and validation accuracy. Success means fewer near-collisions and higher compositional retrieval accuracy at equal dimension; failure is indistinguishable collision statistics or no downstream gain.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.