Unverified 2026

Polygon-Closure Attention

Implementation & benchmark of arXiv:2608.20930 — Mori theory and symplectic reduction for polygons and quivers

Usefulness6/10

Difficulty5/10

Novelty7/10

Geometry Stability Generalization Accuracy Attention Transformer Regularization Embedding Geometry Lie groups Symplectic Representation theory

Source paper: Mori theory and symplectic reduction for polygons and quivers arXiv:2608.20930 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Represent each token value in an attention head as a bounded three-dimensional Lie-algebra vector and impose a weighted polygon-closure condition on the aggregate value vectors. The module is invariant to a common $SU(2)\simeq SO(3)$ rotation of all token vectors, preventing the head from spending capacity on an arbitrary global orientation. A soft closure penalty gives a drop-in experiment, while projection onto the zero-sum manifold provides a harder constrained variant.

Formulas

$$\mathfrak{su}(2)=\left\{A\in M_{2}(\mathbb{C})\mid A^{*}=-A,\ \operatorname{tr}(A)=0\right\},\qquad A=\begin{pmatrix}ia&z\\-\overline z&-ia\end{pmatrix}.$$

$$\langle A,B\rangle=-\frac{1}{2}\operatorname{tr}(AB),\qquad \mu(x_1,\ldots,x_n)=\sum_{i=1}^{n}x_i.$$

$$a_i=\frac{2b_i}{|\mathbf b|},\qquad |\mathbf b|=\sum_{i=1}^{n}b_i,\qquad \mathcal{L}_{\mathrm{close}}=\lambda\frac{\left\|\sum_{i=1}^{n}x_i\right\|_2^2}{\left(\sum_{i=1}^{n}b_i\right)^2},\qquad \|x_i\|_2=b_i.$$

$$x_i=\frac{b_i u_i}{\max(\|u_i\|_2,\varepsilon)},\qquad b_i=\frac{\exp(s_i)}{\sum_j\exp(s_j)}B,\qquad b_i\leq (1/2-\delta)B.$$

Mathematical statement

The paper identifies additive polygon spaces with weighted $\mathbb{P}^{1}$ GIT quotients and describes the $SU(2)$ Lie algebra as traceless skew-Hermitian matrices $A=\begin{pmatrix}ia&z\\-\overline z&-ia\end{pmatrix}$, identified with $\mathbb{R}^{3}$ using the invariant inner product $\langle A,B\rangle=-\frac12\operatorname{tr}(AB)$. For the ML adaptation, let $x_i\in\mathbb{R}^{3}$ be the Lie-algebra embedding of token $i$, let $b_i>0$ be its fixed or learned weight, and constrain $\|x_i\|_2=b_i$. The additive moment-map or polygon-closure equation is $\mu(x)=\sum_{i=1}^{n}x_i=0$; it is invariant under the simultaneous rotation $x_i\mapsto Rx_i$, with $R\in SO(3)$. Use the soft penalty $\mathcal{L}_{\mathrm{close}}=\lambda\|\sum_i x_i\|_2^2/(\sum_i b_i)^2$. A stable radius parameterization is $x_i=b_i u_i/\max(\|u_i\|_2,\varepsilon)$, where $u_i\in\mathbb{R}^{3}$ is produced by the network. The paper normalizes weights as $a_i=2b_i/|\mathbf b|$, where $|\mathbf b|=\sum_i b_i$; its half-total-weight stability boundary motivates preventing one token from receiving nearly all the mass.

Implementation notes

Integrate this into the value branch of one Transformer attention head, not into the query-key dot product initially. For an input sequence $h_1,\ldots,h_n\in\mathbb{R}^{d}$, use a small MLP $f:\mathbb{R}^{d}\to\mathbb{R}^{3}$ to produce $u_i=f(h_i)$, and a second scalar head to produce logits $s_i$. Convert the logits to bounded weights with $b_i=B\,\mathrm{softmax}(s)_i$, using fixed $B=1$ at first. Normalize each vector with $x_i=b_i u_i/(\|u_i\|_2+\varepsilon)$. Compute ordinary attention weights $\alpha_{ij}=\mathrm{softmax}_j(q_i^{\top}k_j/\sqrt d)$, and augment each value by concatenating the three-dimensional geometric feature: $\widetilde v_i=[W_vh_i;x_i]$. Apply the usual output projection to $\sum_i\alpha_{ij}\widetilde v_i$. Add $\lambda(t)\|\sum_i x_i\|_2^2/B^2$ to the training loss, warming $\lambda(t)$ linearly from zero to its target over the first 10% of training. A harder variant applies an approximate projection after forming the vectors: $x_i\leftarrow x_i-\eta_c(\sum_jx_j)/n$, followed by renormalization to radius $b_i$. The mathematical quantities computed exactly are the fixed-radius constraint, the zero moment map, and the simultaneous $SO(3)$-invariant norm. No group constants need estimation; tune $\lambda$, $\eta_c$, $B$, and $\delta$ empirically. To test the half-weight boundary, log $\max_i b_i/B$ and either clip it below $1/2-\delta$ or compare clipped and unclipped models. First run a 2-layer, 8-head Transformer on WikiText-2 or a CIFAR-10 character-level language-modeling task, comparing ordinary attention, soft-penalty attention, and correction-step attention at matched parameter count and FLOPs. Measure training loss variance across seeds, gradient-norm spikes, validation perplexity, and closure residual $\|\sum_i x_i\|/B$. Success means lower seed-to-seed instability or better validation perplexity at equal compute, with the residual decreasing as $\lambda$ increases without optimization collapse.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.