Additive-energy sparse offset design
Implementation & benchmark of arXiv:2607.13666 — Spectral and Additive Combinatorial Methods for Cycles and Absorbing Sets in Lifted-Product Quantum LDPC Codes
Source paper: Spectral and Additive Combinatorial Methods for Cycles and Absorbing Sets in Lifted-Product Quantum LDPC Codes arXiv:2607.13666 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Learn or select sparse cyclic convolution or relative-attention offsets whose pairwise differences collide less often modulo the sequence length. The paper's Fourier fourth-power identity turns this combinatorial objective into an FFT-computable differentiable loss, enabling fixed-K sparse patterns with lower aliasing and interference than random offsets.
Formulas
Mathematical statement
For a subset S of Z_ell, define its group-ring polynomial a(x)=sum_{s in S} x^s and evaluate it at omega^j, where omega=exp(2 pi i/ell). Lemma 4.6 gives sum_{j=0}^{ell-1}|a(omega^j)|^4=ell E(S), where E(S)=|{(i,k,p,q) in S^4: i-k+p-q is congruent to 0 modulo ell}| is the additive energy, counting repeated modular difference collisions. For two offset sets S and T with b(x)=sum_{t in T} x^t, the cross identity is sum_j |a(omega^j)|^2 |b(omega^j)|^2=ell E(S,T), where E(S,T)=|{(i,k,p,q): i,k in S, p,q in T, i-k+p-q is congruent to 0 modulo ell}|. Adapt this to soft mask weights q_s in [0,1] by replacing the indicator polynomial with a_q(x)=sum_{s=0}^{ell-1}q_s x^s and minimizing R_E=ell^(-1) sum_j |a_q(omega^j)|^4. The fixed-cardinality constraint sum_s q_s=K prevents the trivial all-zero solution; projection or a straight-through top-K operator produces an actual sparse pattern.
Implementation notes
Use this as a pattern-learning module for either a sparse circular convolution or a Transformer with K learned relative-position offsets per head. Allocate logits u in R^ell for each head or channel group and compute q=K softmax(u), optionally projecting q toward [0,1]. During the forward pass, use the top-K entries of u as active offsets; use soft weights q during the search phase and a straight-through estimator after top-K selection. Compute q_hat=FFT(q), then R_E=ell^(-1) sum_j |q_hat_j|^4, and add lambda_E R_E to the task loss. For several heads, add lambda_C ell^(-1) sum_j |q_hat_j^(h)|^2 |q_hat_j^(h')|^2 to discourage cross-head collisions, which is the soft version of the paper's E(S,T) identity. The FFT identity computes the exact weighted collision objective; the temperature, regularization weights, and rounding schedule are empirical. First test a four-head sparse relative-position Transformer on a modular arithmetic task and WikiText-2 at sequence length 256, comparing random fixed offsets, learned offsets without the penalty, and energy-regularized offsets. Keep K fixed so memory and FLOPs are identical. Inspect achieved additive energy, attention redundancy, validation perplexity, and throughput. Success is lower energy together with improved accuracy at the same K, especially at larger sequence lengths; failure is no task improvement despite lower energy.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.