# QR-gauged TT neural layer

- ID: 273
- Canonical URL: https://synthcore.org/idea/273/qr-gauged-tt-neural-layer
- API JSON: https://synthcore.org/api/idea/273.json
- API Markdown: https://synthcore.org/api/idea/273.md
- Verification status: unverified
- Source: [arXiv:2606.31061](https://arxiv.org/abs/2606.31061)
- Category: stability
- Solves: stability, accuracy, memory
- ML areas: mlp, optimizer, training-dynamics, fine-tuning
- Math tags: tensor-decomposition, linear-algebra, numerical-analysis, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Use TT gauge freedom to enforce orthonormal interfaces during neural-network training, rather than allowing neighboring cores to develop arbitrarily large and small compensating scales. Periodically perform the paper's QR redistribution after gradient updates or local solves; this leaves the represented weight unchanged while improving the conditioning of subsequent core updates.

## Mathematical statement

The paper updates a core by matricizing it as \(\mathbf{G}_k\), computing the thin QR factorization \(\mathbf{G}_k=\mathbf{Q}\mathbf{R}\), reshaping \(\mathbf{Q}\) back into the current core, and absorbing \(\mathbf{R}\) into the right-neighboring core. Here \(\mathbf{Q}\) has orthonormal columns, \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}\), and \(\mathbf{R}\) is upper triangular. This is a gauge transformation: the product \(\mathbf{Q}\mathbf{R}\) equals the original matricized core, so the represented tensor or neural weight is unchanged in exact arithmetic. The resulting left-orthogonal gauge makes the contraction of cores up to position \(k\) numerically normalized. In the local residual expansion, \(f(\mathcal{G}_k)=f(\mathcal{G}_k^\star)-\langle\mathcal{R},\mathcal{D}\rangle+\tfrac12\|\mathcal{D}\|_F^2\), where \(\mathcal{R}\) is the residual at the local optimum and \(\mathcal{D}\) is the output perturbation induced by \(\Delta\mathcal{G}\). Orthogonal interfaces reduce arbitrary scale factors in the local design operator, which should reduce per-core gradient imbalance and improve the conditioning of local least-squares systems.

## Key formulas

- $$\mathbf{G}_{k}=\mathbf{Q}\mathbf{R},\qquad \mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I},\qquad \mathbf{R}\ \text{upper triangular}$$
- $$\mathcal{G}_{k}\leftarrow\operatorname{reshape}(\mathbf{Q}),\qquad \mathcal{G}_{k+1}\leftarrow\mathbf{R}\,\mathcal{G}_{k+1}$$
- $$f(\mathcal{G}_{k})=f(\mathcal{G}_{k}^{\star})-\langle\mathcal{R},\mathcal{D}\rangle+\frac{1}{2}\|\mathcal{D}\|_{\mathrm{F}}^{2}$$
- $$\mathbf{G}_{k,(12)}=\mathbf{Q}\mathbf{R},\qquad \mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_{r_k}$$

## Implementation notes

1. Integration point: apply gauge maintenance to every TT-matrix layer after each optimizer step, or every \(K\) steps for lower overhead. Operate on core parameter tensors, not activations; the transformation should be function-preserving up to floating-point error. For \(G_k\in\mathbb{R}^{r_{k-1}\times I_k\times J_k\times r_k}\), flatten the first three axes into rows and the final rank axis into columns. 2. Pseudocode: after an AdamW or SGD update, for \(k=1\) to \(n-1\), set \(M=\operatorname{reshape}(G_k,(r_{k-1}I_kJ_k,r_k))\); compute thin QR \((Q,R)=\operatorname{qr}(M)\); set \(G_k=\operatorname{reshape}(Q,(r_{k-1},I_k,J_k,r_k))\); reshape the next core so its first axis is \(r_k\), contract \(R\) with that axis, and restore its shape. Use positive-diagonal QR sign normalization to avoid needless sign flips. Optionally perform a reverse sweep with an RQ or LQ factorization to right-orthogonalize cores. 3. Compute QR directly from the paper's algebra. Estimate the benefit empirically using singular values of interface contractions, gradient norms per core, local design-matrix condition numbers, and the relative function difference \(\|W_{before}-W_{after}\|_F/\|W\|_F\). Do not clip \(R\) in the initial experiment because clipping changes the represented layer. 4. First cheap experiment: train TT-parameterized linear and two-layer MLP models on MNIST or CIFAR-10 using identical AdamW settings, comparing no gauge maintenance, QR every step, and QR every 10 steps. Track training loss, validation accuracy, per-core gradient variance, interface singular-value spread, optimizer stability, and function difference before versus after re-gauging. The expected signal is lower gradient imbalance and fewer NaNs or loss spikes at equal parameter count, while the function difference remains near floating-point tolerance.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.