# Rank-One PSD KATA Attention

- ID: 1069
- Canonical URL: https://synthcore.org/idea/1069/rank-one-psd-kata-attention
- API JSON: https://synthcore.org/api/idea/1069.json
- API Markdown: https://synthcore.org/api/idea/1069.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.17419](https://arxiv.org/abs/2607.17419)
- Category: architecture
- Solves: accuracy, memory, scalability
- ML areas: attention, transformer, kv-cache, memory
- Math tags: geometry, linear-algebra, convex-analysis, combinatorics
- Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10

## Idea description

Replace the usual random or elementwise-positive linear-attention feature map with a rank-one positive-semidefinite feature map derived from query and key vectors. For normalized inputs, the resulting kernel is the squared inner product, which is nonnegative and gives a geometrically structured interference pattern that is better suited to associative recall than an arbitrary low-rank feature map.

## Mathematical statement

The paper defines linear attention using a feature map \(\psi:\mathbb{R}^{d}\to\mathbb{R}^{n_\psi}\), with state \(\mathbf S_t=\sum_{i=1}^t\psi(\mathbf k_i)\mathbf v_i^\top\) and normalizer \(\mathbf Z_t=\sum_{i=1}^t\psi(\mathbf k_i)\). Instantiate the rank-one PSD construction as \(\psi(\mathbf x)=\operatorname{vech}(\mathbf x\mathbf x^\top)\), where \(\operatorname{vech}\) vectorizes the upper triangular matrix and scales off-diagonal entries by \(\sqrt{2}\), giving \(\langle\psi(\mathbf x),\psi(\mathbf y)\rangle=\langle\mathbf x,\mathbf y\rangle^2\). Here \(\mathbf q_t,\mathbf k_i\in\mathbb R^d\) are query and key vectors, \(\mathbf v_i\in\mathbb R^{d_v}\) are values, and \(n_\psi=d(d+1)/2\). The causal output is \(\mathbf z_t=(\psi(\mathbf q_t)^\top\mathbf S_t)/(\psi(\mathbf q_t)^\top\mathbf Z_t+\varepsilon)\). Rank-one PSD matrices lie in the PSD cone, which is self-dual under the Frobenius inner product, so every kernel weight \((\mathbf q^\top\mathbf k)^2\) is nonnegative. For unit key directions \(u_i\in\mathbb R^r\), the Welch interference floor is \(\frac{1}{M(M-1)}\sum_{i\ne j}|u_i^\top u_j|^2\ge\frac{M-r}{r(M-1)}\), where \(M\) is the number of stored directions and \(r\) is the projection dimension.

## Key formulas

- $$\mathbf z_t=\frac{\psi(\mathbf q_t)^\top\mathbf S_t}{\psi(\mathbf q_t)^\top\mathbf Z_t},\qquad \mathbf S_t=\sum_{i=1}^{t}\psi(\mathbf k_i)\mathbf v_i^\top,\qquad \mathbf Z_t=\sum_{i=1}^{t}\psi(\mathbf k_i).$$
- $$\psi(\mathbf x)=\operatorname{vech}(\mathbf x\mathbf x^\top),\qquad \langle\psi(\mathbf q),\psi(\mathbf k)\rangle=\operatorname{tr}(\mathbf q\mathbf q^\top\mathbf k\mathbf k^\top)=(\mathbf q^\top\mathbf k)^2\ge 0.$$
- $$\frac{1}{M(M-1)}\sum_{i\ne j}|u_i^\top u_j|^2\ge\frac{M-r}{r(M-1)},\qquad \|u_i\|_2=1,\quad M>r.$$

## Implementation notes

Integrate this idea into the feature-map operation of a causal linear-attention head. First apply RMSNorm to each query and key, optionally project the head dimension from \(d\) to \(r\), and construct \(\phi(x)=\operatorname{vech}(xx^\top)\). Store the symmetric matrix or packed upper-triangular vector rather than duplicating symmetric entries. For each token, compute `phi_k`, update `S = S + outer(phi_k, v)`, update `Z = Z + phi_k`, and return `z = (phi_q @ S) / (phi_q @ Z + eps)`. If the exact \(r(r+1)/2\) feature dimension is too large, use a fixed orthogonal projection from the packed PSD feature into dimension \(p\), and compare this against a random-feature baseline at equal state size. Accumulate \(S\) and \(Z\) in FP32 even when projections and values use BF16 or FP16. The mathematical guarantee supplies nonnegative kernel weights; estimate actual interference by sampling normalized projected feature vectors and computing their mean pairwise squared inner product, then compare it with the Welch bound using dimension \(p\). The first experiment should use a small Transformer on MQAR, associative recall, and repeated-key overwrite, comparing softmax attention, ELU-plus-one linear attention, and PSD features at matched state bytes. Measure recall at training length and 8x-16x extrapolated length, throughput, and memory. Success is higher recall at equal state size with no severe language-model fluency loss.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a causal normalized rank-one PSD linear-attention MVP using packed vech(xx^T) features, plus an ELU-plus-one random-feature baseline at matched state dimension. The kernel identity held with maximum absolute error 2.13e-14, all 5,000 sampled kernel weights were nonnegative, and every Welch sweep exceeded its theoretical lower bound. PSD attention beat the baseline in all 12 toy associative-recall settings, but this does not establish Transformer-scale or long-context gains.

### Mechanism check

- Verdict: Built a causal normalized rank-one PSD linear-attention MVP using packed vech(xx^T) features, plus an ELU-plus-one random-feature baseline at matched state dimension. The kernel identity held with maximum absolute error 2.13e-14, all 5,000 sampled kernel weights were nonnegative, and every Welch sweep exceeded its theoretical lower bound. PSD attention beat the baseline in all 12 toy associative-recall settings, but this does not establish Transformer-scale or long-context gains.
- Confidence: 8/10
- Limitations: Only a small synthetic associative-recall experiment was tested. No trained Transformer, MQAR benchmark, language modeling, long-context extrapolation, throughput, GPU, mixed precision, memory, learned projections, repeated-key overwrite, or exact matched-FLOP analysis was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.854
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/190/bench_report.json)
- [experiment.py](https://synthcore.org/code/190/experiment.py)
- [report.md](https://synthcore.org/code/190/report.md)
- [report_bench_2026-08-30T205557.md](https://synthcore.org/code/190/report_bench_2026-08-30T205557.md)
- [results.json](https://synthcore.org/code/190/results.json)
- [stage2_bench.py](https://synthcore.org/code/190/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/190)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.