# Odd-Harmonic Spherical Positional Encoding

- ID: 2672
- Canonical URL: https://synthcore.org/idea/2672/odd-harmonic-spherical-positional-encoding
- API JSON: https://synthcore.org/api/idea/2672.json
- API Markdown: https://synthcore.org/api/idea/2672.md
- Verification status: unverified
- Source: [arXiv:2608.21587](https://arxiv.org/abs/2608.21587)
- Category: architecture
- Solves: accuracy, stability, generalization
- ML areas: embedding, transformer, attention
- Math tags: harmonic-analysis, geometry, linear-algebra
- Ratings: usefulness 5/10; difficulty 3/10; novelty 3/10

## Idea description

Replace or augment a scalar periodic positional coordinate with a normalized bank of odd Fourier harmonics, keeping every position on the same-radius sphere. The resulting representation has an explicit translation-invariant similarity kernel, allowing the frequency count and spectral weighting to control how sharply attention distinguishes nearby versus distant phases.

## Mathematical statement

The paper defines the symmetric moment curve for an integer k greater than or equal to 0 as g(t) = (1 divided by square root of k plus 1) times (cos(t), sin(t), cos(3t), sin(3t), ..., cos((2k+1)t), sin((2k+1)t)), where t is an angle modulo 2 pi and g(t) lies in the unit sphere in R^(2k+2). Constant norm follows from ||g(t)||_2^2 = (1/(k+1)) times the sum from j=0 to k of [cos^2((2j+1)t) + sin^2((2j+1)t)] = 1. For two angles t and s, the induced inner-product kernel is K_k(t,s) = g(t)^T g(s) = (1/(k+1)) times the sum from j=0 to k of cos((2j+1)(t-s)) = sin(2(k+1)(t-s)) divided by [2(k+1) sin(t-s)] when sin(t-s) is nonzero, with continuous limiting value 1 at t=s. The Euclidean distance is ||g(t)-g(s)||_2^2 = 2 - 2K_k(t,s). The neural-network adaptation uses nonnegative spectral weights a_j and Z = sum from j=0 to k of a_j^2: p_k(t) = (1/sqrt(Z))(a_0 cos(t), a_0 sin(t), ..., a_k cos((2k+1)t), a_k sin((2k+1)t)). It remains unit norm and has kernel K_a(t,s) = (1/Z) times the sum from j=0 to k of a_j^2 cos((2j+1)(t-s)).

## Key formulas

- $$g(t)=\frac{1}{\sqrt{k+1}}\bigl(\cos t,\sin t,\cos 3t,\sin 3t,\ldots,\cos((2k+1)t),\sin((2k+1)t)\bigr)$$
- $$g(t)^{\mathsf T}g(s)=\frac{1}{k+1}\sum_{j=0}^{k}\cos\bigl((2j+1)(t-s)\bigr)=\frac{\sin(2(k+1)(t-s))}{2(k+1)\sin(t-s)}$$
- $$\|g(t)-g(s)\|_2^2=2-2g(t)^{\mathsf T}g(s)$$
- $$p_k(t)=\frac{1}{\sqrt{Z}}\bigl(a_0\cos t,a_0\sin t,\ldots,a_k\cos((2k+1)t),a_k\sin((2k+1)t)\bigr),\quad Z=\sum_{j=0}^{k}a_j^2,\quad K_a(t,s)=\frac{1}{Z}\sum_{j=0}^{k}a_j^2\cos((2j+1)(t-s))$$

## Implementation notes

(1) Integrate this at the positional-embedding stage of a small decoder-only transformer. For each token position n, map it to a phase t_n = 2*pi*n/P when the task has known period P. For ordinary sequence positions, use t_n = 2*pi*n/P_max with a large P_max, or concatenate this encoding with the existing positional encoding. Construct k+1 harmonic pairs at odd frequencies 1, 3, ..., 2k+1, then project the resulting vector from dimension 2(k+1) to the model dimension d_model using a learned matrix W_p and add it to token embeddings before the first attention block. Start with fixed a_j = 1. An optional learned version uses unconstrained parameters b_j, sets a_j = softplus(b_j), and recomputes Z = sum_j a_j^2 on every forward pass. (2) Pseudocode: for each position n, set t = 2*pi*n/P; initialize z as an empty vector; for j from 0 through k, set omega = 2*j+1 and append a[j]*cos(omega*t) and a[j]*sin(omega*t); divide z by sqrt(sum_j a[j]^2); set h[n] = token_embedding[n] + W_p times z. (3) Compute the embedding and normalization exactly from the displayed formulas. Log the empirical positional kernel and verify that its value before projection equals K_a(t_n,t_m). Estimate no theorem constants; all quantities are directly computable. (4) First experiment: train a 2-layer, 4-head transformer on modular addition, copy-with-period-P, and synthetic periodic sequence-prediction tasks. Compare learned absolute positions, standard sinusoidal positions with frequencies 1, 2, 4, ..., and this odd-harmonic spherical encoding for k in {1, 3, 7, 15}. Keep projected positional dimension and parameter count comparable. Evaluate validation accuracy at training lengths and substantially longer extrapolation lengths, attention entropy versus phase difference, positional-vector norm variation, gradient-norm variance, and wall-clock overhead. The falsifiable hypothesis is improved periodic extrapolation and more stable positional norms, especially when the test period is represented by the encoding.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.