# Positive Lattice Fourier Features

- ID: 2790
- Canonical URL: https://synthcore.org/idea/2790/positive-lattice-fourier-features
- API JSON: https://synthcore.org/api/idea/2790.json
- API Markdown: https://synthcore.org/api/idea/2790.md
- Verification status: unverified
- Source: [arXiv:2608.28792](https://arxiv.org/abs/2608.28792)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: attention, embedding, transformer, regularization
- Math tags: harmonic-analysis, measure-theory, approximation-theory, probability
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Construct positional or relative-position features as a nonnegative mixture of lattice cosine functions instead of independently signed sinusoidal features. The resulting bias is the Fourier transform of a positive discrete measure with explicitly bounded spectral support, while the mesh and degree can be initialized in the paper's dense-but-controlled frequency regime.

## Mathematical statement

Let Q(x) = sum from j = 0 to r of a_j T_j(x), where T_j is the Chebyshev polynomial, r is the degree, and a_j >= 0. Since T_j(cos(t)) = cos(jt), f(z) = C Q(cos(dz)) = C sum from j = 0 to r of a_j cos(jdz), where C > 0 is a scale and d > 0 is the frequency mesh. This is the Fourier transform of the positive discrete measure mu = C a_0 delta_0 + (C/2) sum from j = 1 to r of a_j(delta_{jd} + delta_{-jd}), whose support lies in [-rd, rd]. The paper proves that under local uniform convergence to an entire F of order below two, with F(0) > 0 and the relevant polynomial zeros in [-1,1), limsup r_k d_k^2 <= 4 sum over gamma_n > A of 1/gamma_n^2, where plus or minus gamma_n are the nonzero real zeros of F. Because the reciprocal-square tail tends to zero, the resulting asymptotic design window is r d -> infinity and r d^2 -> 0, equivalently d^{-1} = o(r) and r = o(d^{-2}). For a neural implementation, normalize the coefficients as p_j = exp(theta_j) / sum_i exp(theta_i), making the learned bias b(Delta) = sum from j = 0 to r of p_j cos(j d Delta) a positive cosine expectation.

## Key formulas

- $$Q(x)=\sum_{j=0}^{r}a_jT_j(x),\qquad a_j\ge 0,\qquad T_j(\cos t)=\cos(jt).$$
- $$f(z)=C Q(\cos(dz))=C\sum_{j=0}^{r}a_j\cos(jdz)=\int_{\mathbb R}e^{iz\xi}\,d\mu(\xi),$$
$$\mu=C a_0\delta_0+\frac{C}{2}\sum_{j=1}^{r}a_j(\delta_{jd}+\delta_{-jd}),\qquad \operatorname{supp}(\mu)\subseteq[-rd,rd].$$
- $$\limsup_{k\to\infty}r_kd_k^2\le4\sum_{\gamma_n>A}\frac{1}{\gamma_n^2}.$$
- $$d^{-1}=o(r),\qquad r=o(d^{-2}),\qquad\text{equivalently}\qquad rd\to\infty,\quad rd^2\to0.$$

## Implementation notes

1. Integration point: replace a standard relative-position Fourier bias or random Fourier-feature similarity inside a small Transformer attention block. For token positions u and v, define Delta = u - v and compute b(Delta) = sum_j p_j cos(j d Delta), then add alpha b(Delta) to the attention logits, where alpha is a learned scalar or per-head scale. 2. Pseudocode: choose degree r and initialize d = r^{-beta} with 0.5 < beta < 1; initialize trainable logits theta in R^(r+1); on every forward pass compute p = softmax(theta), construct B[u,v] = sum_{j=0}^r p[j] cos(j*d*(u-v)), and use softmax(QK^T/sqrt(head_dim) + alpha*B). Optionally learn log d, but constrain it or add penalties lambda_1 max(0, epsilon - r*d)^2 + lambda_2 max(0, r*d^2 - eta)^2. 3. The exact mathematical structure computed from the paper is coefficient positivity and spectral support bounded by r*d. The quantities estimated empirically are the effective bandwidth and concentration of p; define B95 = d times the smallest m such that sum from j = 0 to m of p_j is at least 0.95. 4. First experiment: train a 2-layer, 4-head Transformer on a synthetic copy/reversal task and a small character-level WikiText-2 or enwik8 subset. Compare against sinusoidal positions, random Fourier features, and learned relative-position bias at matched parameter count and FLOPs. Test r in {32, 64, 128} and d = r^{-0.75}. Measure validation loss, extrapolation at twice the training sequence length, gradient-norm variance, and attention-logit overflow. The hypothesis is improved length extrapolation and lower optimization variance at equal compute; reject the method if it fails to beat sinusoidal features on extrapolation or stability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.