Tensorized concentration mixing MVP

Implementation

tensorized_mixing.py builds the discrete analogue P_A F^* P_B F P_A, forms its factors, applies the tensor operator one axis at a time without materializing the Kronecker product, and compares against a dense reference.

Quantitative mechanism checks

Results are in results.json (seed 1441).

1. Positive-contraction prediction: every factor has spectrum in [0,1], and the tensor operator has largest eigenvalue equal to the product of factor largest eigenvalues, hence at most 1. Across dimensions 1--5, observed global maxima were 1.0 within 2.7e-15; minimum eigenvalues were zero up to 1.6e-16 roundoff.
2. Exact separable application prediction: sequential axis application equals the explicit Kronecker matrix-vector product. Relative errors for dimensions 2--4 were 1.7e-16, 2.3e-16, and 4.2e-16.
3. Schatten multiplicativity prediction: for p=0.5, 1, and 2, relative errors were at most 3.4e-8, 2.3e-16, and 1.8e-16, respectively. The slightly larger p=0.5 error is numerical conditioning from the quasi-norm calculation.
4. Scaling prediction: for an n-by-n grid, dense matvec work is N^2, while axis-wise work is N*sum(n_j). Measured predicted dense/axis operation ratios were 4.0 (d=2, N=64), 21.33 (d=3, N=512), and 31.25 (d=4, N=625). Matrix parameter reductions were 32x, 1365x, and 3906x in those cases.

Baseline comparison

At 144 tokens, dense storage was 20,736 matrix entries versus 288 factor entries, and both constructions had largest singular value 1.0. The dense implementation was faster for this tiny CPU case (22 microseconds versus 101 microseconds) because dense BLAS amortizes overhead better; this is not evidence against the asymptotic arithmetic reduction. No accuracy/training comparison was run.

Reproduction

/home/maxwelhelp/main/bin/python3 tensorized_mixing.py > results.json

Limitations

This is a NumPy toy verification, not a CIFAR/ViT experiment. It uses fixed Fourier bands rather than learned factors, does not measure GPU memory, and does not test end-to-end accuracy or feature-channel projections. Dense timing was limited to small matrices to keep the verification cheap.