{
 "artifacts": null,
 "category": "architecture",
 "description": "Replace or augment standard sinusoidal or RoPE position features with bracket-quadratic phases $e(-\\theta n\\lfloor\\beta n\\rfloor)$ generated by a Heisenberg nilmanifold orbit. Multiple irrational coefficients and output frequencies produce a cheap deterministic encoding whose empirical cross-position correlations should exhibit cancellation instead of the periodic aliasing of rational or finite-frequency encodings.",
 "formulas_latex": [
  "$$G_1=\\left\\{\\begin{pmatrix}1\u0026x\u0026z\\\\0\u00261\u0026y\\\\0\u00260\u00261\\end{pmatrix}:x,y,z\\in\\mathbb{R}\\right\\},\\qquad F_1\\left(\\begin{pmatrix}1\u0026x\u0026z\\\\0\u00261\u0026y\\\\0\u00260\u00261\\end{pmatrix}\\Gamma_1\\right)=e\\left(z-x\\lfloor y\\rfloor\\right),\\quad e(t)=e^{2\\pi i t}.$$",
  "$$\\begin{pmatrix}1\u0026x\u0026z\\\\0\u00261\u0026y\\\\0\u00260\u00261\\end{pmatrix}\\Gamma_1\\ \\sim\\ \\begin{pmatrix}1\u0026\\{x\\}\u0026\\{z-x\\lfloor y\\rfloor\\}\\\\0\u00261\u0026\\{y\\}\\\\0\u00260\u00261\\end{pmatrix}\\Gamma_1.$$",
  "$$\\text{If }(g_1(n)\\Gamma_1)_{n\\in[X]}\\text{ is }\\delta_1\\text{-equidistributed and }0\u003c\\delta_1\u003c1/10,\\quad \\mathbb{E}_{n\\in[X]}e(-\\theta n\\lfloor\\beta n\\rfloor)\\ll\\delta_1^{1/2}.$$",
  "$$\\phi_{m,r}(n)=\\left(\\cos\\big(2\\pi\\theta_r n\\lfloor\\beta_m n\\rfloor\\big),\\ \\sin\\big(2\\pi\\theta_r n\\lfloor\\beta_m n\\rfloor\\big)\\right),\\qquad p_n=W\\,[n/d,\\phi_{1,1}(n),\\ldots,\\phi_{M,R}(n)]^\\top.$$"
 ],
 "id": 2864,
 "implementation": "Integrate the encoding at the transformer input, before the first attention layer, or add it to the query/key rotation used by RoPE. For a token at position $n\\in\\{0,\\ldots,L-1\\}$, choose $M$ fixed irrational coefficients $\\beta_m$, for example $\\sqrt[3]{2}$, $\\sqrt{2}$, and $\\sqrt{5}$, and $R$ small output frequencies $\\theta_r$ such as $0.5,1,2$. Compute $u_{m,r}=\\theta_r n\\lfloor\\beta_m n\\rfloor$, then append or concatenate $[\\cos(2\\pi u_{m,r}),\\sin(2\\pi u_{m,r})]$ to the usual position vector. A learned matrix $W\\in\\mathbb{R}^{d\\times(1+2MR)}$ projects the normalized position index and bracket features into the model dimension; initialize $W$ with variance $1/(1+2MR)$. Pseudocode: `for n in range(L): features=[n/L]; for beta in betas: q=floor(beta*n); for theta in thetas: features += [cos(2*pi*theta*n*q), sin(2*pi*theta*n*q)]; pos[n]=W @ features; h[n]=token[n]+pos[n]`. The mathematical quantity supplied by the paper is the bracket phase and the predicted cancellation $|L^{-1}\\sum_n e(-\\theta n\\lfloor\\beta n\\rfloor)|\\lesssim\\sqrt{\\delta_1}$ when the associated orbit is equidistributed. Estimate $\\delta_1$ empirically by binning the fractional coordinates $(\\{x_n\\},\\{y_n\\},\\{z_n-x_n\\lfloor y_n\\rfloor\\})$ if an explicit orbit construction is used, or measure the maximum off-diagonal Gram correlation of the resulting feature matrix. First run a cheap ablation on a 6-layer, 256-dimensional decoder on WikiText-2 or a synthetic modular-copy task with context lengths 512, 2K, and 8K. Compare learned absolute embeddings, sinusoidal encoding, RoPE, and RoPE plus bracket features at equal parameter count. Track validation perplexity, copy accuracy at long distances, and the maximum and mean off-diagonal position Gram correlation. Success means lower long-range perplexity or copy-task error without extra attention FLOPs, together with lower position-feature correlation and less degradation when extrapolating beyond the training context.",
 "math_summary": "The paper uses the Heisenberg group $G_1$ of upper-unitriangular matrices and its integer lattice $\\Gamma_1$. A group element with Mal'cev coordinates $(x,y,z)$ is represented as $\\begin{pmatrix}1\u0026x\u0026z\\\\0\u00261\u0026y\\\\0\u00260\u00261\\end{pmatrix}$. The quotient $G_1/\\Gamma_1$ has the coordinate representative $(\\{x\\},\\{y\\},\\{z-x\\lfloor y\\rfloor\\})$. Define $e(t)=\\exp(2\\pi i t)$ and the observable $F_1(x,y,z)=e(z-x\\lfloor y\\rfloor)$. Lemma 3.1 states that if the orbit $g_1(n)\\Gamma_1$ is $\\delta_1$-equidistributed in $G_1/\\Gamma_1$, with $0\u003c\\delta_1\u003c1/10$, then $\\mathbb{E}_{n\\in[X]}e(-\\theta n\\lfloor\\beta n\\rfloor)\\ll\\delta_1^{1/2}$. Here $n$ is the position index, $\\beta$ is the bracket-polynomial coefficient, $\\theta$ is a real output frequency, $X$ is the sequence length, and $\\delta_1$ measures discrepancy from Haar-uniformity. We adapt the scalar complex phase into a real feature pair $(\\cos(2\\pi\\theta n\\lfloor\\beta n\\rfloor),\\sin(2\\pi\\theta n\\lfloor\\beta n\\rfloor))$ and use the cancellation bound as the design criterion for selecting frequencies and validating low average correlation.",
 "math_tags": [
  "dynamical-systems",
  "harmonic-analysis",
  "geometry",
  "lie-groups"
 ],
 "ml_areas": [
  "attention",
  "transformer",
  "embedding"
 ],
 "paper": {
  "arxiv_id": "2608.30078",
  "arxiv_url": "https://arxiv.org/abs/2608.30078",
  "summary_what_math_gives_to_ml": "The paper develops a concrete Heisenberg-nilmanifold representation of bracket-quadratic phases such as $e(-\\theta n\\lfloor \\beta n\\rfloor)$ and proves that quantitative equidistribution of the corresponding nilmanifold orbit forces exponential-sum cancellation. This gives a mathematically structured alternative to ordinary sinusoidal positional encodings: positions are mapped through a discontinuous, noncommutative phase whose long-range correlations can be bounded rather than learned from scratch. The most direct transfer is a deterministic bracket-polynomial positional encoding for long-context transformers, optionally combined with several irrational frequencies and a learned projection. The expected benefit is reduced positional aliasing and more uniform position-feature correlations at negligible inference cost.",
  "title": "Sets whose differences avoid a bracket quadratic",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 3,
  "novelty": 7,
  "usefulness": 5
 },
 "solves": [
  "accuracy",
  "scalability"
 ],
 "title": "Heisenberg Bracket Positional Encoding",
 "url": "https://synthcore.org/idea/2864/heisenberg-bracket-positional-encoding",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}