# Warm-Started Exact Rank Pruning

- ID: 1352
- Canonical URL: https://synthcore.org/idea/1352/warm-started-exact-rank-pruning
- API JSON: https://synthcore.org/api/idea/1352.json
- API Markdown: https://synthcore.org/api/idea/1352.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.24193](https://arxiv.org/abs/2607.24193)
- Category: architecture
- Solves: memory, scalability, accuracy
- ML areas: mlp, transformer, pruning, fine-tuning
- Math tags: optimization, linear-algebra, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Parameterize a trainable weight update as \(\Delta W=UV^{\top}\) with an excessive initial rank \(r\), and penalize active columns using an exact column \(\ell_{2,0}\) penalty. Increase \(\lambda\) along a warm-started path and hard-delete redundant paired columns, producing an automatically selected rank without training a separate model for every candidate rank. Apply scale balancing after each update so pruning decisions are invariant to reciprocal rescaling of factor pairs.

## Mathematical statement

The paper's model is \(\Phi_{\lambda,\mu}(U,V)=\tfrac12\|X-UV^{\top}\|_F^2+\tfrac\mu2(\|U\|_F^2+\|V\|_F^2)+\lambda(\|U\|_{2,0}+\|V\|_{2,0})\), where \(X\) is the target matrix, \(U\in\mathbb R^{n\times r}\) and \(V\in\mathbb R^{m\times r}\) are factors, \(r\) is an upper-bound rank, \(\mu>0\) controls scale, and \(\|U\|_{2,0}=\#\{j:\|u_j\|_2\ne0\}\) counts nonzero columns. For step size \(\eta\), the exact proximal operator of a column penalty is hard thresholding: \(\operatorname{prox}_{\eta\lambda\|\cdot\|_{2,0}}(z)=0\) if \(\|z\|_2^2\le2\eta\lambda\), and equals \(z\) otherwise. For a shared paired component, use the equivalent joint rule \(u_j=v_j=0\) when \(\|u_j\|_2^2+\|v_j\|_2^2\le4\eta\lambda\). Reciprocal scaling preserves the rank-one product: with \(a_j=\sqrt{\|v_j\|_2/(\|u_j\|_2+\epsilon)}\), \(u'_j=a_ju_j\), \(v'_j=v_j/a_j\), so \(u'_jv_j'^{\top}=u_jv_j^{\top}\) and the two norms become approximately equal.

## Key formulas

- $$\Phi_{\lambda,\mu}(U,V)=\frac{1}{2}\|X-UV^{\top}\|_{F}^{2}+\frac{\mu}{2}\left(\|U\|_{F}^{2}+\|V\|_{F}^{2}\right)+\lambda\left(\|U\|_{2,0}+\|V\|_{2,0}\right).$$
- $$\|U\|_{2,0}=\#\{j\in\{1,\ldots,r\}:\|u_j\|_2\neq 0\},\qquad \operatorname{prox}_{\eta\lambda\|\cdot\|_{2,0}}(z)=\begin{cases}0,&\|z\|_2^2\le 2\eta\lambda,\\z,&\|z\|_2^2>2\eta\lambda.\end{cases}$$
- $$U_{t+1}=\mathcal H_{\eta\lambda_t}\!\left(U_t-\eta\left((U_tV_t^{\top}-W)V_t+\mu U_t\right)\right),\quad V_{t+1}=\mathcal H_{\eta\lambda_t}\!\left(V_t-\eta\left((U_{t+1}V_t^{\top}-W)^{\top}U_{t+1}+\mu V_t\right)\right),$$
- $$a_j=\sqrt{\frac{\|v_j\|_2}{\|u_j\|_2+\epsilon}},\qquad u'_j=a_ju_j,\qquad v'_j=v_j/a_j,\qquad u'_jv_j'^{\top}=u_jv_j^{\top}.$$

## Implementation notes

(1) Integration point: use this in a factorized linear layer or LoRA adapter. For a frozen base weight \(W_0\), represent the trainable update as \(\Delta W=UV^\top\), with \(U\in\mathbb R^{d_{out}\times r_{max}}\), \(V\in\mathbb R^{d_{in}\times r_{max}}\), and initialize \(r_{max}\) above the expected useful rank. The forward pass is \(y=W_0x+U(V^\top x)\). Add the quadratic scale penalty and perform paired column pruning during training. (2) Pseudocode: initialize \(U,V\), choose \(0=\lambda_0<\lambda_1<\cdots<\lambda_K\), and warm-start every stage from the previous stage. For each minibatch, compute \(G_U=((UV^\top-W)V)+\mu U\) and \(G_V=((UV^\top-W)^\top U)+\mu V\), where \(W\) is the target matrix in a reconstruction experiment or the task-gradient equivalent in end-to-end training. Set \(Z_U=U-\eta G_U\), \(Z_V=V-\eta G_V\). For each column \(j\), set both columns to zero if \(\|Z_{U,:,j}\|_2^2+\|Z_{V,:,j}\|_2^2\le4\eta\lambda_k\); otherwise retain both. Balance retained pairs with the displayed \(a_j\). (3) Compute exactly the hard threshold, shared active mask, balancing map, and warm-start path from the paper. Estimate no spectra. (4) First experiment: fine-tune a small ViT or CIFAR-10 MLP using LoRA. Compare fixed-rank 16/32/64 LoRA and magnitude-pruned LoRA against \(r_{max}=64\) with a geometric \(\lambda\)-path, increasing \(\lambda\) by 1.5 every 1,000 updates. Select the smallest rank within 0.2 percentage points of the unregularized rank-64 validation accuracy. Measure accuracy, active rank, trainable parameters, wall-clock time, and inference latency. Success means near-baseline accuracy at lower rank, with a stable decreasing rank path and reduced memory or latency.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a NumPy MVP implementing paired exact hard thresholding, reciprocal scale balancing, warm-started lambda continuation, and a fixed-rank reconstruction baseline. The mechanism manifested: the proximal rule had 0 mismatches over 408 swept cases, balancing preserved products to 1.27e-15 maximum error and reduced norm gaps to 8.88e-16, and the lambda path reduced rank monotonically from 8 to 3 while retaining essentially unchanged reconstruction loss. This is a positive toy-setting result, not evidence of an end-to-end task win.

### Mechanism check

- Verdict: Built a NumPy MVP implementing paired exact hard thresholding, reciprocal scale balancing, warm-started lambda continuation, and a fixed-rank reconstruction baseline. The mechanism manifested: the proximal rule had 0 mismatches over 408 swept cases, balancing preserved products to 1.27e-15 maximum error and reduced norm gaps to 8.88e-16, and the lambda path reduced rank monotonically from 8 to 3 while retaining essentially unchanged reconstruction loss. This is a positive toy-setting result, not evidence of an end-to-end task win.
- Confidence: 9/10
- Limitations: Only a small synthetic unconstrained matrix reconstruction problem was tested; no NMF nonnegativity, LoRA/MLP fine-tuning, noisy data, CUDA execution, inference latency, memory measurement, or broad statistical seed sweep was evaluated. The observed rank transition is partly enabled by SVD initialization with deliberately weak surplus columns, so generalization to arbitrary neural-network initialization remains unverified.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.62745
- Paired wins: 2/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/531/bench_report.json)
- [experiment.py](https://synthcore.org/code/531/experiment.py)
- [report.md](https://synthcore.org/code/531/report.md)
- [report_bench_2026-08-31T170519.md](https://synthcore.org/code/531/report_bench_2026-08-31T170519.md)
- [results.json](https://synthcore.org/code/531/results.json)
- [run_output.json](https://synthcore.org/code/531/run_output.json)
- [stage2_bench.py](https://synthcore.org/code/531/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/531)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.