# Polygon-Closure Attention

- ID: 2638
- Canonical URL: https://synthcore.org/idea/2638/polygon-closure-attention
- API JSON: https://synthcore.org/api/idea/2638.json
- API Markdown: https://synthcore.org/api/idea/2638.md
- Verification status: unverified
- Source: [arXiv:2608.20930](https://arxiv.org/abs/2608.20930)
- Category: geometry
- Solves: stability, generalization, accuracy
- ML areas: attention, transformer, regularization, embedding
- Math tags: geometry, lie-groups, symplectic, representation-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Represent each token value in an attention head as a bounded three-dimensional Lie-algebra vector and impose a weighted polygon-closure condition on the aggregate value vectors. The module is invariant to a common \(SU(2)\simeq SO(3)\) rotation of all token vectors, preventing the head from spending capacity on an arbitrary global orientation. A soft closure penalty gives a drop-in experiment, while projection onto the zero-sum manifold provides a harder constrained variant.

## Mathematical statement

The paper identifies additive polygon spaces with weighted \(\mathbb{P}^{1}\) GIT quotients and describes the \(SU(2)\) Lie algebra as traceless skew-Hermitian matrices \(A=\begin{pmatrix}ia&z\\-\overline z&-ia\end{pmatrix}\), identified with \(\mathbb{R}^{3}\) using the invariant inner product \(\langle A,B\rangle=-\frac12\operatorname{tr}(AB)\). For the ML adaptation, let \(x_i\in\mathbb{R}^{3}\) be the Lie-algebra embedding of token \(i\), let \(b_i>0\) be its fixed or learned weight, and constrain \(\|x_i\|_2=b_i\). The additive moment-map or polygon-closure equation is \(\mu(x)=\sum_{i=1}^{n}x_i=0\); it is invariant under the simultaneous rotation \(x_i\mapsto Rx_i\), with \(R\in SO(3)\). Use the soft penalty \(\mathcal{L}_{\mathrm{close}}=\lambda\|\sum_i x_i\|_2^2/(\sum_i b_i)^2\). A stable radius parameterization is \(x_i=b_i u_i/\max(\|u_i\|_2,\varepsilon)\), where \(u_i\in\mathbb{R}^{3}\) is produced by the network. The paper normalizes weights as \(a_i=2b_i/|\mathbf b|\), where \(|\mathbf b|=\sum_i b_i\); its half-total-weight stability boundary motivates preventing one token from receiving nearly all the mass.

## Key formulas

- $$\mathfrak{su}(2)=\left\{A\in M_{2}(\mathbb{C})\mid A^{*}=-A,\ \operatorname{tr}(A)=0\right\},\qquad A=\begin{pmatrix}ia&z\\-\overline z&-ia\end{pmatrix}.$$
- $$\langle A,B\rangle=-\frac{1}{2}\operatorname{tr}(AB),\qquad \mu(x_1,\ldots,x_n)=\sum_{i=1}^{n}x_i.$$
- $$a_i=\frac{2b_i}{|\mathbf b|},\qquad |\mathbf b|=\sum_{i=1}^{n}b_i,\qquad \mathcal{L}_{\mathrm{close}}=\lambda\frac{\left\|\sum_{i=1}^{n}x_i\right\|_2^2}{\left(\sum_{i=1}^{n}b_i\right)^2},\qquad \|x_i\|_2=b_i.$$
- $$x_i=\frac{b_i u_i}{\max(\|u_i\|_2,\varepsilon)},\qquad b_i=\frac{\exp(s_i)}{\sum_j\exp(s_j)}B,\qquad b_i\leq (1/2-\delta)B.$$

## Implementation notes

Integrate this into the value branch of one Transformer attention head, not into the query-key dot product initially. For an input sequence \(h_1,\ldots,h_n\in\mathbb{R}^{d}\), use a small MLP \(f:\mathbb{R}^{d}\to\mathbb{R}^{3}\) to produce \(u_i=f(h_i)\), and a second scalar head to produce logits \(s_i\). Convert the logits to bounded weights with \(b_i=B\,\mathrm{softmax}(s)_i\), using fixed \(B=1\) at first. Normalize each vector with \(x_i=b_i u_i/(\|u_i\|_2+\varepsilon)\). Compute ordinary attention weights \(\alpha_{ij}=\mathrm{softmax}_j(q_i^{\top}k_j/\sqrt d)\), and augment each value by concatenating the three-dimensional geometric feature: \(\widetilde v_i=[W_vh_i;x_i]\). Apply the usual output projection to \(\sum_i\alpha_{ij}\widetilde v_i\). Add \(\lambda(t)\|\sum_i x_i\|_2^2/B^2\) to the training loss, warming \(\lambda(t)\) linearly from zero to its target over the first 10% of training. A harder variant applies an approximate projection after forming the vectors: \(x_i\leftarrow x_i-\eta_c(\sum_jx_j)/n\), followed by renormalization to radius \(b_i\). The mathematical quantities computed exactly are the fixed-radius constraint, the zero moment map, and the simultaneous \(SO(3)\)-invariant norm. No group constants need estimation; tune \(\lambda\), \(\eta_c\), \(B\), and \(\delta\) empirically. To test the half-weight boundary, log \(\max_i b_i/B\) and either clip it below \(1/2-\delta\) or compare clipped and unclipped models. First run a 2-layer, 8-head Transformer on WikiText-2 or a CIFAR-10 character-level language-modeling task, comparing ordinary attention, soft-penalty attention, and correction-step attention at matched parameter count and FLOPs. Measure training loss variance across seeds, gradient-norm spikes, validation perplexity, and closure residual \(\|\sum_i x_i\|/B\). Success means lower seed-to-seed instability or better validation perplexity at equal compute, with the residual decreasing as \(\lambda\) increases without optimization collapse.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.