Greedy Tensor-CUR Weight Compression

(1) Integration point: replace a dense linear or convolutional weight tensor after pretraining, or initialize a compressed layer before training. For an MLP, reshape a matrix \(W\in\mathbb{R}^{d_{out}\times d_{in}}\) into balanced modes such as \((d_{out,1},d_{out,2},d_{in,1},d_{in,2})\). For a convolution, retain output-channel, input-channel, height, and width modes. Execute the forward pass through Tucker contractions involving \(U_1,\ldots,U_N\) and the small core \(G\). (2) Pseudocode: choose target ranks \(R_n\); initialize index sets \(J_n\) using largest-norm fibers or leverage-score sampling; form \(C_n=W_{(n)}(:,J_n)\); compute \(U_n=\operatorname{orth}(C_n)\); form \(S=W(J_1,\ldots,J_N)\); compute \(G=S\times_n U_n(J_n,:)^{\dagger}\) for every mode; reconstruct \(\widehat W\); evaluate entry or fiber residuals on a held-out subset; add the index associated with the largest residual fiber and repeat until the target ranks are reached. (3) Computed from the mathematics: QR orthogonalization, small pseudoinverses, sampled core, Tucker contractions, and Frobenius/max-entry reconstruction errors. Estimated empirically: residual-driven index choices, rank allocation, conditioning thresholds, and fine-tuning duration. Reject or resample a factor if its smallest singular value is below \(10^{-5}\) times its largest singular value. (4) First experiment: compress the two largest linear layers of CIFAR-10 ViT-Tiny or ResNet-18, comparing dense weights, standard Tucker-SVD, random TCUR, and greedy TCUR at 25%, 12.5%, and 6.25% parameter budgets. Fine-tune each model for 10 epochs with identical optimization settings. Measure accuracy, parameter memory, multiply-add count, wall-clock inference latency, relative Frobenius error, and maximum-entry error. Success means greedy TCUR matches or exceeds standard Tucker accuracy at equal memory and produces lower latency; an initial target is 2x parameter reduction with less than one percentage point accuracy loss.