All-Cut Schatten Control for Polynomial Layers

1. Integration point: add this to a tensorized polynomial or multilinear MLP layer whose weight is stored as \(K\) with mode dimensions \((d_1,\ldots,d_m,d_{in},d_{out})\). The forward pass can be \(y=K\times_1 x_1\cdots\times_m x_m\), or a randomized version with independent Gaussian feature vectors \(g_j\) replacing the stochastic modes. Compute the profile penalty only during training; do not change the forward graph initially. 2. Pseudocode: initialize a cut set \(\mathcal Q\) with the empty cut, full cut, and 4--8 random subsets of \([m]\). Every \(T\) optimizer steps, reshape \(K\) for each \(S\in\mathcal Q\) into a matrix with rows \(\mathcal H_{S^c}\otimes\mathcal E\) and columns \(\mathcal H_S\otimes\mathcal C\); estimate its top \(k\) singular values by randomized SVD; set \(P=\max_S(\sum_{i\le k}\sigma_i^r)^{1/r}\); add \(\lambda[\log(P/\tau)]_+^2\) to the task loss. On a violation, add the currently worst cut to \(\mathcal Q\). Optionally rescale \(K\leftarrow K\min(1,\tau/P)\) after each optimizer update. 3. The paper supplies the profile-to-chaos inequality and the \((p+r)^{m/2}\) dependence; the singular values, worst cut, and profile are estimated numerically. Use \(r=2\) first, then compare \(r=4\) and a spectral surrogate \(r=\infty\). 4. First experiment: train order-2 and order-3 tensorized MLPs on CIFAR-10 or a synthetic polynomial regression task, comparing Frobenius decay, spectral normalization, and profile regularization at equal parameter count and FLOPs. Measure activation-norm tail percentiles, gradient norm variance, loss stability across width, and test error. Success is a lower 99th-percentile activation/gradient norm and fewer unstable runs without sacrificing validation accuracy; a stronger result is stable scaling to larger mode widths at fixed \(\lambda\) and improved test error at equal parameter count.