Aligned-Orthogonal Tensor Layer

1. Integration point: replace a dense MLP weight matrix or convolutional kernel with a tensorized weight. For an MLP matrix $W\in\mathbb{R}^{D_{out}\times D_{in}}$, reshape it into a $d$-way tensor with mode sizes whose products equal the input and output dimensions, or use a four-way tensor and contract it with a reshaped input. Store factor matrices $U^{(m)}\in\mathbb{R}^{n_m\times R}$ and coefficients $c\in\mathbb{R}^{R}$. 2. Pseudocode: initialize factors using an HOSVD-like decomposition or random QR; at each forward pass compute the contraction $y=\operatorname{contract}(\sum_r c_r u_r^{(1)}\otimes\cdots\otimes u_r^{(d)},x)$. After each optimizer step, either apply QR retraction $U^{(m)}\leftarrow\operatorname{qf}(U^{(m)})$ and absorb the triangular factors into $c$, or add the displayed orthogonality penalty. Every $K$ steps, sort $|c_r|$, mask the smallest terms, and optionally regrow masked terms using residual gradients. 3. Computed from the paper's mathematics: shared rank-one indexing, mode-wise orthogonality, coefficient truncation, and low-bato-rank compression. Estimated empirically: the useful rank $R$, pruning schedule, penalty $\lambda$, and whether strict QR or soft orthogonality works better. 4. First experiment: train a small Transformer encoder or CIFAR-10 MLP with one tensorized projection, testing $R\in\{0.25R_{max},0.5R_{max},0.75R_{max}\}$. Compare dense, unconstrained CP, Tucker, post-hoc SVD, and aligned-orthogonal versions at matched parameter counts. Measure validation accuracy, parameters, forward latency, gradient norm variance, and loss at equal optimizer steps. Success means at least 2x parameter reduction with less than one percentage point accuracy loss, or better stability and accuracy than unconstrained CP at the same parameter count.