# Local Krylov-TT residual block

- ID: 478
- Canonical URL: https://synthcore.org/idea/478/local-krylov-tt-residual-block
- API JSON: https://synthcore.org/api/idea/478.json
- API Markdown: https://synthcore.org/api/idea/478.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.06453](https://arxiv.org/abs/2607.06453)
- Category: architecture
- Solves: memory, scalability, speedup
- ML areas: mlp, ssm, graph-nn, memory
- Math tags: tensor-decomposition, linear-algebra, numerical-analysis, dynamical-systems
- Ratings: usefulness 7/10; difficulty 6/10; novelty 5/10

## Idea description

Represent a high-order feature tensor as a tensor train and replace a dense global feature transform by a truncated polynomial in a learned nearest-neighbor operator. The block computes a short Krylov expansion, p_m(A)x = sum from k=0 to m of c_k A^k x, compressing back to a fixed TT rank after each operator application; locality is intended to prevent rank growth from scaling with the total number of tensor sites.

## Mathematical statement

The paper studies linear equations and ODEs on the tensor-product space V = tensor product from mu=1 to d of V_mu, with A u = b and du/dt = A u, u(0) = u_0. A tensor-train representation writes a tensor x in tensor product from mu=1 to d of V_mu as x(i_1,...,i_d) = G_1[i_1] G_2[i_2] ... G_d[i_d], where i_mu indexes the local basis of V_mu, G_mu[i_mu] is an r_(mu-1) by r_mu matrix, r_0 = r_d = 1, and r_mu are TT ranks. The relevant operator structure is a sum of terms acting on neighboring sites, A = A_0 + sum over mu=1,...,d-1 and ell=1,...,s of A_(mu,ell)^(loc) tensor-product A_(mu+1,ell)^(loc), with identity factors on all other sites. Krylov methods use the polynomial space K_m(A,x) = span{x, Ax, ..., A^m x}, so the neural adaptation is p_m(A)x = sum from k=0 to m of c_k A^k x. The paper's rank-increasing analysis uses locality and partial commutativity to argue that nearest-neighbor operator polynomials can have ranks controlled mainly by polynomial degree, local operator rank, and approximation tolerance, rather than by the total number of sites. The engineering hypothesis is that TT rounding after each local operator application preserves this advantage during neural forward and backward passes.

## Key formulas

- $$\mathcal{V}=\bigotimes_{\mu=1}^{d}V_{\mu}$$
- $$\mathbf{A}\mathbf{u}=\mathbf{b}$$
- $$\frac{d}{dt}\mathbf{u}=\mathbf{A}\mathbf{u},\qquad \mathbf{u}(0)=\mathbf{u}_{0}$$
- $$p_m(A)x=\sum_{k=0}^{m}c_kA^kx,\qquad \mathcal{K}_m(A,x)=\operatorname{span}\{x,Ax,\ldots,A^mx\}$$

## Implementation notes

(1) Integration point: replace the wide linear layer inside a residual MLP block. Reshape a feature vector of length n^d into a d-way tensor with local width n, store the hidden state in TT format with ranks at most r, and define a learned nearest-neighbor operator A = A_site + sum over mu and ell of L_ell^(mu) tensor-product M_ell^(mu+1), with identity factors on nonadjacent sites. The local factors may be shared across sites or generated from a small site embedding. (2) Pseudocode: input TT tensor x, coefficients c_0 through c_m, and operator factors A. Set z = x and y = c_0 x. For k from 1 to m, compute z = A z by contracting each local Kronecker term with neighboring TT cores, add the resulting TT tensors, then apply TT rounding z = Round(z,r,epsilon). Accumulate y = y + c_k z, round y to rank r, and return x plus alpha times the inverse reshape of LayerNorm(y). The polynomial/Krylov formula appears in the accumulation line, while TT rounding occurs after every A application rather than only at the end. (3) Compute from the mathematics: exact local contractions, TT addition, and SVD-based rounding. Estimate empirically: choose c_k by learning them directly, or initialize them with a Chebyshev approximation to a target scalar filter on an estimated spectral interval. Estimate the interval with 10 to 20 power or Lanczos iterations on random TT vectors. Track maximum intermediate rank before rounding and discarded Frobenius energy. (4) First experiment: train a d=8, n=4 synthetic classification task whose target is generated by a dense random linear map followed by a nonlinear head. Compare a dense MLP, a standard TT-linear layer with the same parameter count, and this local Krylov-TT block for m in {2,4,6} and r in {4,8,16}. Then test a tensorized MLP on CIFAR-10. Success means equal validation accuracy with substantially fewer parameters and activation memory, or faster loss descent at equal parameter count. Use a concrete target of at least two times lower tensorized-layer memory with no more than one percentage point accuracy loss. Also measure whether peak TT rank stays approximately flat as d increases, directly testing the locality-based premise.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built an exact TT-SVD/Krylov prototype with nearest-neighbor operator application, per-step rounding, polynomial identity verification, rank/coupling/degree/locality sweeps, and a small regression comparison. The mechanism manifested: zero coupling retained rank 1, degree 1-4 produced ranks 3,5,7,9, and local ranks plateaued at 7 for d=8-12 while dense degree-1 ranks grew 4,8,16. The direct polynomial identity error was 0.0; the toy regression also favored the local polynomial features, but this comparison used an exactly matched target and is only secondary evidence.

### Mechanism check

- Verdict: Built an exact TT-SVD/Krylov prototype with nearest-neighbor operator application, per-step rounding, polynomial identity verification, rank/coupling/degree/locality sweeps, and a small regression comparison. The mechanism manifested: zero coupling retained rank 1, degree 1-4 produced ranks 3,5,7,9, and local ranks plateaued at 7 for d=8-12 while dense degree-1 ranks grew 4,8,16. The direct polynomial identity error was 0.0; the toy regression also favored the local polynomial features, but this comparison used an exactly matched target and is only secondary evidence.
- Confidence: 8/10
- Limitations: No neural-network training, CIFAR-10 evaluation, GPU implementation, backward-pass benchmark, activation-memory measurement, learned operator factors, or statistically repeated seeds were tested. The dense comparison is feasible only for d<=8, and the toy regression target is deliberately generated by the same local polynomial family, so it does not establish a general accuracy or speed win.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0163
- Paired wins: 1/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/236/bench_report.json)
- [krylov_tt_experiment.py](https://synthcore.org/code/236/krylov_tt_experiment.py)
- [report.md](https://synthcore.org/code/236/report.md)
- [report_bench_2026-08-30T212319.md](https://synthcore.org/code/236/report_bench_2026-08-30T212319.md)
- [results.json](https://synthcore.org/code/236/results.json)
- [stage2_bench.py](https://synthcore.org/code/236/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/236)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.