# Greedy Tensor-CUR Weight Compression

- ID: 1160
- Canonical URL: https://synthcore.org/idea/1160/greedy-tensor-cur-weight-compression
- API JSON: https://synthcore.org/api/idea/1160.json
- API Markdown: https://synthcore.org/api/idea/1160.md
- Verification status: unverified
- Source: [arXiv:2607.19741](https://arxiv.org/abs/2607.19741)
- Category: memory
- Solves: memory, inference-speedup, accuracy
- ML areas: mlp, cnn, pruning, inference-speedup
- Math tags: tensor-decomposition, linear-algebra, approximation-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 5/10

## Idea description

Compress a dense neural-network weight tensor with mode-wise selected fibers and a small sampled core, then convert the result into a Tucker layer for efficient execution. Select indices greedily using reconstruction residuals rather than random sampling, and fine-tune the compressed model to recover task accuracy.

## Mathematical statement

The paper defines the tensor maximum-entry norm and Frobenius norm as \(\|\mathcal{A}\|_{\max}=\max_{i_1,\ldots,i_N}|a_{i_1\ldots i_N}|\) and \(\|\mathcal{A}\|_F=(\sum_{i_1=1}^{I_1}\cdots\sum_{i_N=1}^{I_N}|a_{i_1\ldots i_N}|^2)^{1/2}\), giving explicit reconstruction-error metrics. It uses orthonormal mode factors \(Q_n\in\mathbb{R}^{I_n\times R_n}\) satisfying \(Q_n^\top Q_n=I_{R_n}\), which improves numerical stability by preventing arbitrary factor scaling. For a weight tensor \(W\in\mathbb{R}^{I_1\times\cdots\times I_N}\), let \(J_n\subset\{1,\ldots,I_n\}\) contain selected indices, let \(C_n=W_{(n)}(:,J_n)\) be selected mode-n fibers, and let \(S=W(J_1,\ldots,J_N)\) be the intersection subtensor. A practical TCUR-to-Tucker approximation is \(\widehat W=G\times_1U_1\times_2\cdots\times_NU_N\), with \(U_n=\operatorname{orth}(C_n)\) and \(G=S\times_1U_1(J_1,:)^{\dagger}\times_2\cdots\times_NU_N(J_N,:)^{\dagger}\). The pseudoinverses act only on small sampled matrices, while the optimization target is low relative Frobenius error and controlled maximum-entry error.

## Key formulas

- $$\|\mathcal{A}\|_{\max}=\max_{\substack{i_n=1,2,\dots,I_n\\n=1,2,\dots,N}}|a_{i_1i_2\dots i_N}|,\qquad \|\mathcal{A}\|_F=\sqrt{\sum_{i_1=1}^{I_1}\cdots\sum_{i_N=1}^{I_N}|a_{i_1i_2\dots i_N}|^2}.$$
- $$Q_n^\top Q_n=I_{R_n},\qquad Q_n\in\mathbb{R}^{I_n\times R_n}.$$
- $$\widehat W=G\times_1U_1\times_2\cdots\times_NU_N,\qquad U_n=\operatorname{orth}\!\left(W_{(n)}(:,J_n)\right),$$
- $$G=S\times_1U_1(J_1,:)^{\dagger}\times_2\cdots\times_NU_N(J_N,:)^{\dagger},\qquad S=W(J_1,\ldots,J_N).$$

## Implementation notes

(1) Integration point: replace a dense linear or convolutional weight tensor after pretraining, or initialize a compressed layer before training. For an MLP, reshape a matrix \(W\in\mathbb{R}^{d_{out}\times d_{in}}\) into balanced modes such as \((d_{out,1},d_{out,2},d_{in,1},d_{in,2})\). For a convolution, retain output-channel, input-channel, height, and width modes. Execute the forward pass through Tucker contractions involving \(U_1,\ldots,U_N\) and the small core \(G\). (2) Pseudocode: choose target ranks \(R_n\); initialize index sets \(J_n\) using largest-norm fibers or leverage-score sampling; form \(C_n=W_{(n)}(:,J_n)\); compute \(U_n=\operatorname{orth}(C_n)\); form \(S=W(J_1,\ldots,J_N)\); compute \(G=S\times_n U_n(J_n,:)^{\dagger}\) for every mode; reconstruct \(\widehat W\); evaluate entry or fiber residuals on a held-out subset; add the index associated with the largest residual fiber and repeat until the target ranks are reached. (3) Computed from the mathematics: QR orthogonalization, small pseudoinverses, sampled core, Tucker contractions, and Frobenius/max-entry reconstruction errors. Estimated empirically: residual-driven index choices, rank allocation, conditioning thresholds, and fine-tuning duration. Reject or resample a factor if its smallest singular value is below \(10^{-5}\) times its largest singular value. (4) First experiment: compress the two largest linear layers of CIFAR-10 ViT-Tiny or ResNet-18, comparing dense weights, standard Tucker-SVD, random TCUR, and greedy TCUR at 25%, 12.5%, and 6.25% parameter budgets. Fine-tune each model for 10 epochs with identical optimization settings. Measure accuracy, parameter memory, multiply-add count, wall-clock inference latency, relative Frobenius error, and maximum-entry error. Success means greedy TCUR matches or exceeds standard Tucker accuracy at equal memory and produces lower latency; an initial target is 2x parameter reduction with less than one percentage point accuracy loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.