Online TT-ALS for Streaming Tensor Decomposition with Incremental Orthogonalization
arXiv:2606.31061
2026
Optimization
2 ideas extracted · analyzed Aug 29, 2026
What the math gives to ML
The paper provides a constructive streaming update for tensor-train factors: solve each core exactly by least squares, then QR-orthogonalize it and absorb the triangular factor into the neighboring core. The transferable asset is the combination of gauge fixing, exact local minimization, and a residual identity showing that each local update is non-increasing in reconstruction error. This suggests replacing gradient updates for low-rank neural layers or adapters with streaming TT-ALS updates on minibatches of activation-target pairs, while using orthogonal cores to improve conditioning and prevent factor scale drift. The most credible first target is a frozen-feature linear or MLP projection, where the local least-squares problems are explicit and can be compared directly against LoRA and AdamW.
Ideas from this paper
Unverified
Re-invented
2026
Parameterize a frozen-feature neural projection as a tensor-train operator and update one TT core at a time by an exact least-squares solve on each incoming minibatch. After every core solve, QR-orthogonalize its matricization and absorb the triangular factor into the next core, preserving the represented operator while controlling conditioning. This creates a deterministic, low-memory alternative to Adam-trained LoRA for regression heads, MLP projections, or linear attention projections.
Useful8/10
Difficulty6/10
Novelty6/10
Unverified
Re-invented
2026
Use TT gauge freedom to enforce orthonormal interfaces during neural-network training, rather than allowing neighboring cores to develop arbitrarily large and small compensating scales. Periodically perform the paper's QR redistribution after gradient updates or local solves; this leaves the represented weight unchanged while improving the conditioning of subsequent core updates.
Useful6/10
Difficulty4/10
Novelty7/10