# Graded Levy-area recurrent state

- ID: 2011
- Canonical URL: https://synthcore.org/idea/2011/graded-levy-area-recurrent-state
- API JSON: https://synthcore.org/api/idea/2011.json
- API Markdown: https://synthcore.org/api/idea/2011.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.08756](https://arxiv.org/abs/2608.08756)
- Category: architecture
- Solves: accuracy, scalability, memory
- ML areas: rnn, ssm, embedding
- Math tags: algebra, geometry, linear-algebra
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace additive recurrent pooling with a graded state containing the current feature increment, an antisymmetric order-sensitive area matrix, and an optional symmetric quadratic-variation accumulator. Compose chunks using the paper's exact group law, allowing a sequence model to retain compressed pairwise ordering information without explicitly forming all token pairs.

## Mathematical statement

The paper represents a finite second-order state as X=(x,A,Q), where x is a first-order vector, A is an antisymmetric operator or matrix satisfying A^*=-A, and Q is a symmetric operator satisfying Q^*=Q. Its sequential composition is (x,A,Q) star (y,B,R)=(x+y, A+B+1/2(x wedge_op y), Q+R), with x wedge_op y := x tensor y - y tensor x. Thus the area update adds the cross-interaction between two consecutive chunks, while Q is additive. The dilation delta_lambda(x,A,Q)=(lambda x,lambda^2 A,lambda^2 Q) states that first-order and second-order channels should use different normalization scales. The representation is useful because A changes sign when the order of x and y is reversed, unlike a symmetric covariance or sum.

## Key formulas

- $$X=\frac{1}{2}x^{\otimes 2}+A-\frac{1}{2}Q,\qquad A^{*}=-A,\qquad Q^{*}=Q,$$
- $$x\wedge_{\mathrm{op}}y:=x\otimes y-y\otimes x,$$
- $$(x,A,Q)\star(y,B,R)=\left(x+y,\,A+B+\frac{1}{2}x\wedge_{\mathrm{op}}y,\,Q+R\right),$$
- $$\delta_{\lambda}(x,A,Q)=(\lambda x,\lambda^{2}A,\lambda^{2}Q).$$

## Implementation notes

Integrate this at the recurrent or state-space memory update, before the usual gated transition or output projection. Let each token embedding h_t in R^d be converted into a bounded increment x_t=P tanh(W h_t), where P projects to a small state dimension k, typically 16-64. Maintain s_t=(u_t,A_t,Q_t), with u_t in R^k, A_t in R^{k x k} constrained to be skew-symmetric, and Q_t either diagonal or low-rank symmetric. For a chunk increment (v,B,R), use the exact update u_new=u+v, A_new=A+B+0.5*(u v^T-v u^T), and Q_new=Q+R. In the simplest token-level version set v=x_t, B=0, and R=diag(x_t squared); in a chunked version compute B and R recursively inside each chunk and merge chunks with the same formula. Feed [u_new, vec_skew(A_new), diag(Q_new)] through LayerNorm and a learned linear projection to produce the recurrent output. Normalize u by sqrt(t), A by t, and Q by t, matching the dilation law that second-order coordinates scale quadratically. Use a rank-r factorization A=UV^T-VU^T when k is larger, or maintain only a random/sketched bilinear map to avoid O(k^2) memory. Pseudocode: initialize u,A,Q=0; for each x, compute v=encoder(x), update A=A+0.5*(outer(u,v)-outer(v,u)), update u=u+v, update Q=Q+diag(v*v), and read out concat(u/sqrt(t), A/t, Q/t). The mathematical update is exact; the empirical choices are projection dimension, rank, gating, and normalization. First test on permuted-MNIST, ListOps, and a small character-level language-model benchmark using a GRU, diagonal SSM, and additive pooling as baselines. Compare equal hidden-state bytes and equal FLOPs. Success is higher accuracy or perplexity at the same memory, particularly on tasks where AB and BA must produce different outputs; also verify that chunked inference matches token-by-token inference up to floating-point error.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built the graded Levy-area state with exact skew-symmetric area, diagonal quadratic variation, dilation-compatible normalization, and chunk composition. All quantitative mechanism checks matched prediction exactly: reversal area scaled as |a|/2 and canceled under reversal, repeated-pair area accumulated as n/2, dilation ratios were 1.0 for u, A, and Q, and chunking error was 2.7e-15. The small benchmark did not show an accuracy win: Levy and additive pooling both reached 0.512, while Levy was slower and used more parameters.

### Mechanism check

- Verdict: Built the graded Levy-area state with exact skew-symmetric area, diagonal quadratic variation, dilation-compatible normalization, and chunk composition. All quantitative mechanism checks matched prediction exactly: reversal area scaled as |a|/2 and canceled under reversal, repeated-pair area accumulated as n/2, dilation ratios were 1.0 for u, A, and Q, and chunking error was 2.7e-15. The small benchmark did not show an accuracy win: Levy and additive pooling both reached 0.512, while Levy was slower and used more parameters.
- Confidence: 9/10
- Limitations: Only a tiny synthetic order-sensitive classification task was tested, not permuted-MNIST, ListOps, or language modeling. The benchmark used full-batch training and did not equalize hidden-state bytes or FLOPs; CUDA cuDNN failed on the shared device, so the reported benchmark used CPU fallback.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.21995
- Paired wins: 7/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/815/bench_report.json)
- [experiment.py](https://synthcore.org/code/815/experiment.py)
- [report.md](https://synthcore.org/code/815/report.md)
- [report_bench_2026-09-01T111645.md](https://synthcore.org/code/815/report_bench_2026-09-01T111645.md)
- [results.json](https://synthcore.org/code/815/results.json)
- [stage2_bench.py](https://synthcore.org/code/815/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/815)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.