Idea description

Replace a dense mixing or attention matrix on tokens arranged on a Cartesian grid by a product of learned or fixed one-dimensional concentration operators. The layer applies one axis operator at a time, reducing parameter and compute cost while enforcing that the global operator is a positive contraction with controlled spectral leakage.

Formulas

$$S=P_AQ_BP_A,\qquad Q_B=\mathcal{F}^{-1}P_B\mathcal{F},\qquad 0\preceq S\preceq I.$$

$$S=\bigotimes_{j=1}^{d}S_j,\qquad S_j=P_{A_j}Q_{B_j}P_{A_j},\quad A=\prod_{j=1}^dA_j,\quad B=\prod_{j=1}^dB_j.$$

$$\bigotimes_{j=1}^d A_j-\bigotimes_{j=1}^d B_j=\sum_{k=1}^d\left(\bigotimes_{j<k}A_j\right)\otimes(A_k-B_k)\otimes\left(\bigotimes_{j>k}B_j\right).$$

$$\left\|\bigotimes_{j=1}^dT_j\right\|_{S_p}=\prod_{j=1}^d\|T_j\|_{S_p},\qquad \left\|\sum_kT_k\right\|_{S_p}^{p}\leq\sum_k\|T_k\|_{S_p}^{p}\quad(0<p\leq1).$$

Mathematical statement

The paper studies S=P_AQ_BP_A, where P_A is multiplication by the indicator of a spatial set A and Q_B=\mathcal{F}^{-1}P_B\mathcal{F} is Fourier localization to a frequency set B. For product boxes A=\prod_{j=1}^d A_j and B=\prod_{j=1}^d B_j, the corresponding separable operator has tensor form S=\bigotimes_{j=1}^d S_j, with S_j=P_{A_j}Q_{B_j}P_{A_j}. Each S_j is self-adjoint and obeys 0\preceq S_j\preceq I, so the product also satisfies 0\preceq S\preceq I. The telescoping identity $\bigotimes_j A_j-\bigotimes_j B_j=\sum_k(\bigotimes_{j<k}A_j)\otimes(A_k-B_k)\otimes(\bigotimes_{j>k}B_j)$ isolates one-dimensional discrepancies. Schatten norms multiply across tensor factors, $\|\bigotimes_jT_j\|_{S_p}=\prod_j\|T_j\|_{S_p}$, and for 0<p\leq1 the quasi-triangle inequality bounds the sum. The transferable asset is separable application with a global contraction and multiplicative leakage control.

Implementation notes

Integrate this at the spatial-mixing or attention operation for tokens on a Cartesian grid, such as image patches, video patches, or a diffusion latent grid. Let the feature tensor have shape [batch, channels, n1, ..., nd]. Create one matrix S_j of shape [n_j,n_j] per axis, either learned or initialized from a one-dimensional Fourier concentration kernel. Parameterize each factor as S_j=U_j diag(sigmoid(theta_j)) U_j^T, where U_j has orthonormal columns; this guarantees eigenvalues in [0,1]. Apply the global operator without materializing its Kronecker product: set Y=X, then for k=1,...,d reshape Y so axis k is the matrix-multiplication dimension, multiply each slice by S_k, and reshape back. Add a pointwise value projection and residual connection, for example Y = X + V(X) + mode_matmul(...mode_matmul(X,S_1,1)...,S_d,d). For an interval frequency band [beta_j^-, beta_j^+], initialize the kernel using exp(pi*i*(x_a-x_b)*(beta_j^-+beta_j^+))*sin(pi*(x_a-x_b)*(beta_j^+-beta_j^-))/(pi*(x_a-x_b)), with diagonal beta_j^+-beta_j^+. The paper supplies the tensor factorization, telescoping decomposition, and Schatten multiplicativity; contraction is enforced directly by the parameterization. First test on CIFAR-10 with a small 2D ViT using a 14x14 token grid. Compare dense attention, axial attention, and factorized mixing at matched accuracy and FLOPs. Measure wall-clock time, peak memory, validation accuracy, and largest singular value. Success means lower memory and axis-wise scaling with equal accuracy and no exploding activations.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a discrete Fourier concentration MVP with sequential tensorized application and dense Kronecker references. The mechanism manifested: contraction held to about 3e-15, tensor application matched dense multiplication at 1e-16 relative error, and Schatten multiplicativity matched within 3.4e-8. Parameter and arithmetic scaling matched the predicted reductions, although tiny-matrix wall-clock timing favored dense BLAS due to overhead; no end-to-end accuracy win was tested.

Agent confidence: 9/10

Baseline: At 144 tokens: 20,736 dense parameters; 22.1 microseconds per CPU matvec; largest singular value 1.0000000000000009.
Idea: At 144 tokens: 288 factor parameters, 72x fewer; 101.3 microseconds sequential CPU application; largest singular value 1.0000000000000009. Predicted dense/axis operation ratios were 4.0, 21.33, and 31.25 for the tested 2D/3D/4D sweeps.

Limitations: Only a NumPy toy experiment was run. Learned factors, GPU memory/speed, CIFAR-10 or ViT accuracy, training stability, channel projections, and realistic large-grid performance were not tested.

How to run: python3 tensorized_mixing.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.26361) Sep 2, 2026

Beats the tuned baseline by 31.3% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.2863
Idea mean: 0.1967
Effect (Δ): -0.0896 (−31.3%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±14.1%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

The benchmark used the built-in sequence forecast track only. The local factorized model replaces the standard attention sublayer with a simpler learned symmetric mixer, so although embedding, positional parameters, depth, feed-forward width, head, optimizer, epochs, batch size, and learning-rate search were matched, exact parameter/FLOP parity was not audited. GPU memory, wall-clock speed, scaling with grid dimensions, dense-vs-factorized runtime, and generalization to image/video Cartesian grids were not measured; the mechanism signature was collected from one trained idea model (seed 0).

How to run: python3 bench_tensorized.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Worked ✓

Agent confidence: 7/10

Built a local tensorized concentration mixer for the structurally matching multi-token sequence forecast task and compared it against a dense transformer-style baseline using the shared train_model path, matched learning-rate union, and 8 paired seeds. The idea achieved lower test MSE on all 8 seeds: 0.19666 versus 0.28626, paired delta -0.08960 with permutation p=0.0081, so the bench verdict is idea better (significant). The trained factors had maximum singular value 0.5104, below the predicted bound 1.0; however, the residual block itself is not a contraction and no wall-clock/FLOP or peak-memory measurement was included.

Baseline: Sequence test MSE: mean 0.2862644270, std 0.0536852991; best lr 0.001; per-seed [0.2573727667, 0.2441381812, 0.2655849457, 0.2142851949, 0.3280071318, 0.2741968632, 0.3978331685, 0.3086971641].
Idea: Sequence test MSE: mean 0.1966622919, std 0.0332916755; best lr 0.003; all 8 paired wins; delta -0.0896021351; permutation p=0.0081; observed maximum factor singular value 0.5103706717.

Benchmark result

Beats the tuned baseline by 31.3% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.2863
Idea mean: 0.1967
Effect (Δ): -0.0896 (−31.3%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±14.1%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

The benchmark used the built-in sequence forecast track only. The local factorized model replaces the standard attention sublayer with a simpler learned symmetric mixer, so although embedding, positional parameters, depth, feed-forward width, head, optimizer, epochs, batch size, and learning-rate search were matched, exact parameter/FLOP parity was not audited. GPU memory, wall-clock speed, scaling with grid dimensions, dense-vs-factorized runtime, and generalization to image/video Cartesian grids were not measured; the mechanism signature was collected from one trained idea model (seed 0).

How to run: python3 bench_tensorized.py

Artifacts

Implementation overview ⬇ Download all as ZIP 7 files · code, reports and structured results

📄 REPORT.md 2.2 KB View
📊 bench_report.json 3.0 KB View Raw
🐍 bench_tensorized.py 5.3 KB View Raw
📄 report.md 1.4 KB View
📄 report_bench_2026-09-02T222002.md 5.3 KB View
📊 results.json 3.6 KB View Raw
🐍 tensorized_mixing.py 4.3 KB View Raw

Idea description

Formulas

Mathematical statement

Implementation notes

Verification

Artifacts

Related ideas