△ Mechanism confirmed, baseline not beaten 2026

Local Krylov-TT residual block

Implementation & benchmark of arXiv:2607.06453 — On low-rank tensor train approximability for linear nearest neighbor systems

Usefulness7/10

Difficulty6/10

Novelty5/10

Architecture Memory Scalability Speedup Mlp Ssm Graph nn Memory Tensor decomposition Linear algebra Numerical analysis Dynamical systems

Source paper: On low-rank tensor train approximability for linear nearest neighbor systems arXiv:2607.06453 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Represent a high-order feature tensor as a tensor train and replace a dense global feature transform by a truncated polynomial in a learned nearest-neighbor operator. The block computes a short Krylov expansion, p_m(A)x = sum from k=0 to m of c_k A^k x, compressing back to a fixed TT rank after each operator application; locality is intended to prevent rank growth from scaling with the total number of tensor sites.

Formulas

$$\mathcal{V}=\bigotimes_{\mu=1}^{d}V_{\mu}$$

$$\mathbf{A}\mathbf{u}=\mathbf{b}$$

$$\frac{d}{dt}\mathbf{u}=\mathbf{A}\mathbf{u},\qquad \mathbf{u}(0)=\mathbf{u}_{0}$$

$$p_m(A)x=\sum_{k=0}^{m}c_kA^kx,\qquad \mathcal{K}_m(A,x)=\operatorname{span}\{x,Ax,\ldots,A^mx\}$$

Mathematical statement

The paper studies linear equations and ODEs on the tensor-product space V = tensor product from mu=1 to d of V_mu, with A u = b and du/dt = A u, u(0) = u_0. A tensor-train representation writes a tensor x in tensor product from mu=1 to d of V_mu as x(i_1,...,i_d) = G_1[i_1] G_2[i_2] ... G_d[i_d], where i_mu indexes the local basis of V_mu, G_mu[i_mu] is an r_(mu-1) by r_mu matrix, r_0 = r_d = 1, and r_mu are TT ranks. The relevant operator structure is a sum of terms acting on neighboring sites, A = A_0 + sum over mu=1,...,d-1 and ell=1,...,s of A_(mu,ell)^(loc) tensor-product A_(mu+1,ell)^(loc), with identity factors on all other sites. Krylov methods use the polynomial space K_m(A,x) = span{x, Ax, ..., A^m x}, so the neural adaptation is p_m(A)x = sum from k=0 to m of c_k A^k x. The paper's rank-increasing analysis uses locality and partial commutativity to argue that nearest-neighbor operator polynomials can have ranks controlled mainly by polynomial degree, local operator rank, and approximation tolerance, rather than by the total number of sites. The engineering hypothesis is that TT rounding after each local operator application preserves this advantage during neural forward and backward passes.

Implementation notes

(1) Integration point: replace the wide linear layer inside a residual MLP block. Reshape a feature vector of length n^d into a d-way tensor with local width n, store the hidden state in TT format with ranks at most r, and define a learned nearest-neighbor operator A = A_site + sum over mu and ell of L_ell^(mu) tensor-product M_ell^(mu+1), with identity factors on nonadjacent sites. The local factors may be shared across sites or generated from a small site embedding. (2) Pseudocode: input TT tensor x, coefficients c_0 through c_m, and operator factors A. Set z = x and y = c_0 x. For k from 1 to m, compute z = A z by contracting each local Kronecker term with neighboring TT cores, add the resulting TT tensors, then apply TT rounding z = Round(z,r,epsilon). Accumulate y = y + c_k z, round y to rank r, and return x plus alpha times the inverse reshape of LayerNorm(y). The polynomial/Krylov formula appears in the accumulation line, while TT rounding occurs after every A application rather than only at the end. (3) Compute from the mathematics: exact local contractions, TT addition, and SVD-based rounding. Estimate empirically: choose c_k by learning them directly, or initialize them with a Chebyshev approximation to a target scalar filter on an estimated spectral interval. Estimate the interval with 10 to 20 power or Lanczos iterations on random TT vectors. Track maximum intermediate rank before rounding and discarded Frobenius energy. (4) First experiment: train a d=8, n=4 synthetic classification task whose target is generated by a dense random linear map followed by a nonlinear head. Compare a dense MLP, a standard TT-linear layer with the same parameter count, and this local Krylov-TT block for m in {2,4,6} and r in {4,8,16}. Then test a tensorized MLP on CIFAR-10. Success means equal validation accuracy with substantially fewer parameters and activation memory, or faster loss descent at equal parameter count. Use a concrete target of at least two times lower tensorized-layer memory with no more than one percentage point accuracy loss. Also measure whether peak TT rank stays approximately flat as d increases, directly testing the locality-based premise.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built an exact TT-SVD/Krylov prototype with nearest-neighbor operator application, per-step rounding, polynomial identity verification, rank/coupling/degree/locality sweeps, and a small regression comparison. The mechanism manifested: zero coupling retained rank 1, degree 1-4 produced ranks 3,5,7,9, and local ranks plateaued at 7 for d=8-12 while dense degree-1 ranks grew 4,8,16. The direct polynomial identity error was 0.0; the toy regression also favored the local polynomial features, but this comparison used an exactly matched target and is only secondary evidence.

Agent confidence: 8/10

Baseline: Dense degree-1 transform TT ranks: d=4/6/8 -> 4/8/16; tiny regression test MSE 2.8140006974582152e-09 with 64 input parameters.
Idea: Local Krylov TT ranks for degree 3: d=4/6/8/10/12 -> 4/6/7/7/7; coupling 0 rank 1 versus coupling >=0.1 activating ranks 3/5/7/9 through degree 4; tiny local-polynomial regression test MSE 2.8023441369740366e-12 with 3 polynomial coefficients; exact polynomial accumulation relative error 0.0.

Limitations: No neural-network training, CIFAR-10 evaluation, GPU implementation, backward-pass benchmark, activation-memory measurement, learned operator factors, or statistically repeated seeds were tested. The dense comparison is feasible only for d<=8, and the toy regression target is deliberately generated by the same local polynomial family, so it does not establish a general accuracy or speed win.

How to run: python3 krylov_tt_experiment.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06453) Sep 2, 2026

Significantly WORSE than the tuned baseline (+13.6%, p=0.0163, wins 1 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.4431
Idea mean: 0.5032
Effect (Δ): 0.0601 (+13.6%; negative = idea better)
Wins: 1 / 8 paired seeds
p-value: 0.0163 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±8.4%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in sequence track was tested. CIFAR-10, tabular, dynamics, custom tracks, longer training, explicit TT-SVD tensor compression, activation-memory measurement, FLOP benchmarking, and larger model widths were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented and trained a local Krylov residual block on the structurally matched sequence forecast track, using a matched global token-mixing baseline, identical training budgets, parity-complete hyperparameter settings, and 8 paired seeds. The trained-model mechanism signature confirmed greater locality, but the idea had significantly worse test MSE, so it did not win the benchmark.

Baseline: Best lr=0.003, alpha=0.5; full mean MSE=0.4431248903, std=0.0605058033.
Idea: Best lr=0.003, alpha=0.5; full mean MSE=0.5032319352, std=0.0589014486; paired delta=+0.0601070449; permutation p=0.0163; idea wins 1/8.
Mechanism signature: Baseline trained-model far/near gradient ratio=2.9092408897; idea=1.2671472982; confirmed=true.

Benchmark result