✗ Failed on benchmark 2026

Rank-One PSD KATA Attention

Implementation & benchmark of arXiv:2607.17419 — Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

Usefulness8/10

Difficulty6/10

Novelty7/10

Architecture Accuracy Memory Scalability Attention Transformer Kv cache Memory Geometry Linear algebra Convex analysis Combinatorics

Source paper: Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones arXiv:2607.17419 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the usual random or elementwise-positive linear-attention feature map with a rank-one positive-semidefinite feature map derived from query and key vectors. For normalized inputs, the resulting kernel is the squared inner product, which is nonnegative and gives a geometrically structured interference pattern that is better suited to associative recall than an arbitrary low-rank feature map.

Formulas

$$\mathbf z_t=\frac{\psi(\mathbf q_t)^\top\mathbf S_t}{\psi(\mathbf q_t)^\top\mathbf Z_t},\qquad \mathbf S_t=\sum_{i=1}^{t}\psi(\mathbf k_i)\mathbf v_i^\top,\qquad \mathbf Z_t=\sum_{i=1}^{t}\psi(\mathbf k_i).$$

$$\psi(\mathbf x)=\operatorname{vech}(\mathbf x\mathbf x^\top),\qquad \langle\psi(\mathbf q),\psi(\mathbf k)\rangle=\operatorname{tr}(\mathbf q\mathbf q^\top\mathbf k\mathbf k^\top)=(\mathbf q^\top\mathbf k)^2\ge 0.$$

$$\frac{1}{M(M-1)}\sum_{i\ne j}|u_i^\top u_j|^2\ge\frac{M-r}{r(M-1)},\qquad \|u_i\|_2=1,\quad M>r.$$

Mathematical statement

The paper defines linear attention using a feature map $\psi:\mathbb{R}^{d}\to\mathbb{R}^{n_\psi}$, with state $\mathbf S_t=\sum_{i=1}^t\psi(\mathbf k_i)\mathbf v_i^\top$ and normalizer $\mathbf Z_t=\sum_{i=1}^t\psi(\mathbf k_i)$. Instantiate the rank-one PSD construction as $\psi(\mathbf x)=\operatorname{vech}(\mathbf x\mathbf x^\top)$, where $\operatorname{vech}$ vectorizes the upper triangular matrix and scales off-diagonal entries by $\sqrt{2}$, giving $\langle\psi(\mathbf x),\psi(\mathbf y)\rangle=\langle\mathbf x,\mathbf y\rangle^2$. Here $\mathbf q_t,\mathbf k_i\in\mathbb R^d$ are query and key vectors, $\mathbf v_i\in\mathbb R^{d_v}$ are values, and $n_\psi=d(d+1)/2$. The causal output is $\mathbf z_t=(\psi(\mathbf q_t)^\top\mathbf S_t)/(\psi(\mathbf q_t)^\top\mathbf Z_t+\varepsilon)$. Rank-one PSD matrices lie in the PSD cone, which is self-dual under the Frobenius inner product, so every kernel weight $(\mathbf q^\top\mathbf k)^2$ is nonnegative. For unit key directions $u_i\in\mathbb R^r$, the Welch interference floor is $\frac{1}{M(M-1)}\sum_{i\ne j}|u_i^\top u_j|^2\ge\frac{M-r}{r(M-1)}$, where $M$ is the number of stored directions and $r$ is the projection dimension.

Implementation notes

Integrate this idea into the feature-map operation of a causal linear-attention head. First apply RMSNorm to each query and key, optionally project the head dimension from $d$ to $r$, and construct $\phi(x)=\operatorname{vech}(xx^\top)$. Store the symmetric matrix or packed upper-triangular vector rather than duplicating symmetric entries. For each token, compute phi_k, update S = S + outer(phi_k, v), update Z = Z + phi_k, and return z = (phi_q @ S) / (phi_q @ Z + eps). If the exact $r(r+1)/2$ feature dimension is too large, use a fixed orthogonal projection from the packed PSD feature into dimension $p$, and compare this against a random-feature baseline at equal state size. Accumulate $S$ and $Z$ in FP32 even when projections and values use BF16 or FP16. The mathematical guarantee supplies nonnegative kernel weights; estimate actual interference by sampling normalized projected feature vectors and computing their mean pairwise squared inner product, then compare it with the Welch bound using dimension $p$. The first experiment should use a small Transformer on MQAR, associative recall, and repeated-key overwrite, comparing softmax attention, ELU-plus-one linear attention, and PSD features at matched state bytes. Measure recall at training length and 8x-16x extrapolated length, throughput, and memory. Success is higher recall at equal state size with no severe language-model fluency loss.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not tested

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a causal normalized rank-one PSD linear-attention MVP using packed vech(xx^T) features, plus an ELU-plus-one random-feature baseline at matched state dimension. The kernel identity held with maximum absolute error 2.13e-14, all 5,000 sampled kernel weights were nonnegative, and every Welch sweep exceeded its theoretical lower bound. PSD attention beat the baseline in all 12 toy associative-recall settings, but this does not establish Transformer-scale or long-context gains.

Agent confidence: 8/10

Baseline: ELU-plus-one random projected features: normalized MSE 0.406-0.856 across d={4,8,16}, M={4,8,16,32}.
Idea: Rank-one PSD features: normalized MSE 0.017-0.619; better in 12/12 settings with win fraction 1.0 in every setting; kernel maximum absolute identity error 2.13e-14; minimum sampled kernel 8.73e-08; Welch minimum margin 0.0149.

Limitations: Only a small synthetic associative-recall experiment was tested. No trained Transformer, MQAR benchmark, language modeling, long-context extrapolation, throughput, GPU, mixed precision, memory, learned projections, repeated-key overwrite, or exact matched-FLOP analysis was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.17419) Aug 30, 2026

No significant difference vs. the tuned baseline (+0.3%, p=0.854, wins 3 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.6701
Idea mean: 0.6719
Effect (Δ): 0.0019 (+0.3%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.854 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±4.0%
Mechanism: n/a
Practical effect: No effect
Baseline tuning: swept over 6 configs

Limitations:

Only the built-in sequence forecast track and transformer_tiny were tested. No CIFAR, dynamics, long-context extrapolation, throughput, memory, mixed precision, or larger Transformer evaluation was performed.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Rank-One PSD KATA causal attention in a matched transformer_tiny sequence-forecast system and ran the full 8-seed paired benchmark with equal hyperparameter-union sweeps. The trained-model mechanism prediction was confirmed: observed negative kernel-weight fraction was 0.0, but PSD task MSE was slightly worse than tuned softmax attention and not significant, so no transfer win was demonstrated.

Baseline: Softmax transformer_tiny; best lr=0.003, weight_decay=0.0001; 8-seed test MSE mean 0.6700743958.
Idea: Rank-One PSD transformer_tiny; 8-seed test MSE mean 0.6719429269. Paired delta idea-baseline=+0.0018685311, 3/8 wins, permutation p=0.854.

Benchmark result