Probe-guarded adaptive low-rank layers

1. Integration point: take every large dense MLP or projection weight block $W\in\mathbb{R}^{m\times n}$, partition it into tiles such as $512\times512$, and store each tile as factors $U\in\mathbb{R}^{m\times k}$ and $V\in\mathbb{R}^{k\times n}$ instead of dense weights. At inference compute $xW^T$ as $(xV^T)U^T$; for the first experiment, construct the factors offline and fine-tune $U,V$ directly. 2. Construction pseudocode: choose $d=2(m+n)$ probe pairs using a deterministic permutation-diagonal set, for example $(i,i\bmod n)$ and $(i\bmod m,i)$, and cache the corresponding dense weights. Initialize empty $U,V$. For $k=1,2,\ldots,k_{\max}$, select an unrepresented pivot row and column using the largest currently available cross residual, form ACA factors $u_k=r_{:,j_k}/r_{i_kj_k}$ and $v_k^T=r_{i_k,:}$, where $r$ is the residual cross, then append $u_k,v_k$. Recompute probe predictions $\widehat W_s=U_{i_s,:}V_{:,j_s}$, calculate $e_k^s=W_{i_sj_s}-\widehat W_s$, and stop only after $p=3$ consecutive values of $\widehat\eta_k\leq\varepsilon$. 3. The paper-derived part is the cross-factor representation, linear-number extra entries, and sustained convergence rule; pivot selection, tile size, and probe placement are engineering choices. Estimate actual compression error periodically on a held-out random subset of 10,000 entries to test whether probe error predicts global Frobenius error. 4. First experiment: compress the 4096-dimensional projection and MLP matrices of a small Transformer on WikiText-2 or CIFAR-10, comparing dense weights, fixed-rank SVD, randomized SVD, ordinary ACA, and probe-guarded ACA at equal parameter count. Sweep $\varepsilon\in\{10^{-2},10^{-3},10^{-4}\}$ and report validation loss, rank, weight memory, matrix-vector latency, and the frequency of catastrophic early-stop tiles. Success is lower or equal perplexity at the same memory as fixed-rank SVD, fewer false low-error tiles, and measurable inference speedup; a useful target is 1.5--2x weight-memory reduction with less than 1% relative loss degradation.