△ Mechanism confirmed, baseline not beaten 2026

Warm-Started Exact Rank Pruning

Implementation & benchmark of arXiv:2607.24193 — Automatic Model-Order Selection for Nonnegative Matrix Factorization via Column $\ell_{2,0}$ Regularization

Usefulness7/10

Difficulty5/10

Novelty6/10

Architecture Memory Scalability Accuracy Mlp Transformer Pruning Fine tuning Optimization Linear algebra Dynamical systems

Source paper: Automatic Model-Order Selection for Nonnegative Matrix Factorization via Column $\ell_{2,0}$ Regularization arXiv:2607.24193 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Parameterize a trainable weight update as $\Delta W=UV^{\top}$ with an excessive initial rank $r$, and penalize active columns using an exact column $\ell_{2,0}$ penalty. Increase $\lambda$ along a warm-started path and hard-delete redundant paired columns, producing an automatically selected rank without training a separate model for every candidate rank. Apply scale balancing after each update so pruning decisions are invariant to reciprocal rescaling of factor pairs.

Formulas

$$\Phi_{\lambda,\mu}(U,V)=\frac{1}{2}\|X-UV^{\top}\|_{F}^{2}+\frac{\mu}{2}\left(\|U\|_{F}^{2}+\|V\|_{F}^{2}\right)+\lambda\left(\|U\|_{2,0}+\|V\|_{2,0}\right).$$

$$\|U\|_{2,0}=\#\{j\in\{1,\ldots,r\}:\|u_j\|_2\neq 0\},\qquad \operatorname{prox}_{\eta\lambda\|\cdot\|_{2,0}}(z)=\begin{cases}0,&\|z\|_2^2\le 2\eta\lambda,\\z,&\|z\|_2^2>2\eta\lambda.\end{cases}$$

$$U_{t+1}=\mathcal H_{\eta\lambda_t}\!\left(U_t-\eta\left((U_tV_t^{\top}-W)V_t+\mu U_t\right)\right),\quad V_{t+1}=\mathcal H_{\eta\lambda_t}\!\left(V_t-\eta\left((U_{t+1}V_t^{\top}-W)^{\top}U_{t+1}+\mu V_t\right)\right),$$

$$a_j=\sqrt{\frac{\|v_j\|_2}{\|u_j\|_2+\epsilon}},\qquad u'_j=a_ju_j,\qquad v'_j=v_j/a_j,\qquad u'_jv_j'^{\top}=u_jv_j^{\top}.$$

Mathematical statement

The paper's model is $\Phi_{\lambda,\mu}(U,V)=\tfrac12\|X-UV^{\top}\|_F^2+\tfrac\mu2(\|U\|_F^2+\|V\|_F^2)+\lambda(\|U\|_{2,0}+\|V\|_{2,0})$, where $X$ is the target matrix, $U\in\mathbb R^{n\times r}$ and $V\in\mathbb R^{m\times r}$ are factors, $r$ is an upper-bound rank, $\mu>0$ controls scale, and $\|U\|_{2,0}=\#\{j:\|u_j\|_2\ne0\}$ counts nonzero columns. For step size $\eta$, the exact proximal operator of a column penalty is hard thresholding: $\operatorname{prox}_{\eta\lambda\|\cdot\|_{2,0}}(z)=0$ if $\|z\|_2^2\le2\eta\lambda$, and equals $z$ otherwise. For a shared paired component, use the equivalent joint rule $u_j=v_j=0$ when $\|u_j\|_2^2+\|v_j\|_2^2\le4\eta\lambda$. Reciprocal scaling preserves the rank-one product: with $a_j=\sqrt{\|v_j\|_2/(\|u_j\|_2+\epsilon)}$, $u'_j=a_ju_j$, $v'_j=v_j/a_j$, so $u'_jv_j'^{\top}=u_jv_j^{\top}$ and the two norms become approximately equal.

Implementation notes

(1) Integration point: use this in a factorized linear layer or LoRA adapter. For a frozen base weight $W_0$, represent the trainable update as $\Delta W=UV^\top$, with $U\in\mathbb R^{d_{out}\times r_{max}}$, $V\in\mathbb R^{d_{in}\times r_{max}}$, and initialize $r_{max}$ above the expected useful rank. The forward pass is $y=W_0x+U(V^\top x)$. Add the quadratic scale penalty and perform paired column pruning during training. (2) Pseudocode: initialize $U,V$, choose $0=\lambda_0<\lambda_1<\cdots<\lambda_K$, and warm-start every stage from the previous stage. For each minibatch, compute $G_U=((UV^\top-W)V)+\mu U$ and $G_V=((UV^\top-W)^\top U)+\mu V$, where $W$ is the target matrix in a reconstruction experiment or the task-gradient equivalent in end-to-end training. Set $Z_U=U-\eta G_U$, $Z_V=V-\eta G_V$. For each column $j$, set both columns to zero if $\|Z_{U,:,j}\|_2^2+\|Z_{V,:,j}\|_2^2\le4\eta\lambda_k$; otherwise retain both. Balance retained pairs with the displayed $a_j$. (3) Compute exactly the hard threshold, shared active mask, balancing map, and warm-start path from the paper. Estimate no spectra. (4) First experiment: fine-tune a small ViT or CIFAR-10 MLP using LoRA. Compare fixed-rank 16/32/64 LoRA and magnitude-pruned LoRA against $r_{max}=64$ with a geometric $\lambda$-path, increasing $\lambda$ by 1.5 every 1,000 updates. Select the smallest rank within 0.2 percentage points of the unregularized rank-64 validation accuracy. Measure accuracy, active rank, trainable parameters, wall-clock time, and inference latency. Success means near-baseline accuracy at lower rank, with a stable decreasing rank path and reduced memory or latency.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a NumPy MVP implementing paired exact hard thresholding, reciprocal scale balancing, warm-started lambda continuation, and a fixed-rank reconstruction baseline. The mechanism manifested: the proximal rule had 0 mismatches over 408 swept cases, balancing preserved products to 1.27e-15 maximum error and reduced norm gaps to 8.88e-16, and the lambda path reduced rank monotonically from 8 to 3 while retaining essentially unchanged reconstruction loss. This is a positive toy-setting result, not evidence of an end-to-end task win.

Agent confidence: 9/10

Baseline: Fixed-rank rank-3 loss 1.500000e-6; rank-8 loss 1.500000e-6; 0.434-0.473 s in the initial run and 0.485-0.487 s in the reproducibility run.
Idea: Warm-start path ranks [8,3,3,3,3,3,3,3] for lambdas [0,0.02,0.08,0.2,0.5,1,2,4], loss 1.500000e-6 at every stage, 0.503 s; threshold mismatches 0/408; maximum balancing product error 1.27e-15 and norm gap 8.88e-16.

Limitations: Only a small synthetic unconstrained matrix reconstruction problem was tested; no NMF nonnegativity, LoRA/MLP fine-tuning, noisy data, CUDA execution, inference latency, memory measurement, or broad statistical seed sweep was evaluated. The observed rank transition is partly enabled by SVD initialization with deliberately weak surplus columns, so generalization to arbitrary neural-network initialization remains unverified.

How to run: python3 experiment.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.24193) Sep 2, 2026

No significant difference vs. the tuned baseline (+1.1%, p=0.6274, wins 2 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.1434
Idea mean: 7.2206
Effect (Δ): 0.0771 (+1.1%; negative = idea better)
Wins: 2 / 8 paired seeds
p-value: 0.6274 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±4.5%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 6 configs

Limitations:

Only the registered tabular track was tested; vision, sequence, dynamics, and custom tracks were not tested. The lambda grid was too weak to trigger actual pruning, so automatic neural-network rank reduction was not demonstrated.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Warm-started exact paired rank pruning was implemented and evaluated on the registered tabular Friedman regression track with the shared factorized mlp_tiny system. The tuned baseline achieved MSE 7.14342, while the best idea achieved 7.22056; paired delta_mean was +0.07714 with permutation p=0.62745, hence no significant win. The mechanism signature confirmed monotone rank behavior and near-zero balancing gaps, but no columns were pruned at the tested lambda values.

Baseline: Best cfg {lr: 0.006, weight_decay: 0.0001}; full 8-seed MSE 7.1434198618 ± 0.2740066216.
Idea: Best full 8-seed MSE 7.2205554843 ± 0.3491257387; paired delta +0.0771356225, 2/8 wins, p=0.62745; active rank remained 43 at all stages and pruned_columns=0 for every seed.

Benchmark result