Singular-Tail Budgeted Rank Allocation
Implementation & benchmark of arXiv:2607.01125 — ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
Source paper: ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces arXiv:2607.01125 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the activation singular-value spectrum to allocate a fixed zeroth-order parameter budget across layers instead of assigning the same rank everywhere. Layers with a large discarded singular-value tail receive more coefficient directions, while spectrally compressible layers use smaller adapters, preserving the gradient-relevant subspace under a global memory and query budget.
Formulas
Mathematical statement
For layer l, let X_l in R^{b_l x m_l} have SVD X_l=U_l D_l V_l^T, with singular values sigma_{l,1} >= sigma_{l,2} >= ... . Its exact gradient is g_{W_l}=V_l D_l U_l^T g_{Y_l}. Retaining rank r_l produces the projected gradient V_{l,r_l}V_{l,r_l}^T g_{W_l}, with a worst-case residual bounded by sigma_{l,r_l+1}(X_l)||g_{Y_l}||_F. If the coefficient adapter is Delta W_l=V_{l,r_l}A_l, its parameter and zeroth-order search dimension are d_l=r_l n_l. Given a total budget B, choose integer ranks r_l to minimize a measured tail-error proxy E_l(r_l)=sigma_{l,r_l+1}(X_l)^2 times an estimate of ||g_{Y_l}||_F^2, subject to sum_l r_l n_l <= B. A greedy allocation repeatedly gives one additional rank to the layer with the largest reduction [E_l(r_l)-E_l(r_l+1)]/n_l, approximating the best error reduction per trainable parameter and per ZO dimension.
Implementation notes
1. Integration point: apply this to the rank configuration of activation-SVD ZO adapters; the forward computation remains Y_l=X_lW_l+X_lV_{l,r_l}A_l. The global budget B can be expressed in trainable coefficients, sum_l r_l n_l, or in adapter memory. 2. Calibration: for each candidate layer l, collect representative activations X_l, compute singular values sigma_{l,j} using randomized SVD or eigendecomposition of X_l^T X_l, and record n_l. Estimate q_l by running a few pilot finite-difference directions and using the squared norm of the resulting coefficient gradient as a proxy; alternatively set q_l=1 for a spectrum-only baseline. 3. Allocation pseudocode: initialize every r_l=r_min; while sum_l r_l*n_l+n_l <= B, compute score_l=[sigma_{l,r_l+1}^2-sigma_{l,r_l+2}^2]*q_l/n_l, select the largest score, and increment that r_l. Freeze the resulting bases and train all A_l with the antithetic estimator. Recompute allocation only between experiments, not during training, so the method remains one-shot and forward-only. 4. Compare uniform rank, parameter-count-matched random allocation, and greedy spectral allocation on a 125M model using WikiText-2 or a small instruction dataset. Sweep B over 25%, 50%, and 100% of the uniform-rank budget. Record validation loss, number of trainable coefficients, forward evaluations, and peak memory. The falsifiable signal is that greedy allocation reaches the same validation loss with fewer coefficients or lower loss at fixed B; if singular spectra do not predict downstream performance, the method should fail to beat uniform allocation.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built a self-contained MVP implementing the SVD residual-bound check, gradient-energy proxy, greedy budgeted rank allocator, and matched synthetic regression experiment. The chain-rule identity and residual bound passed numerically, but the greedy allocator selected the same [2,2] ranks as uniform allocation and achieved identical test MSE (0.58904); the matched random [1,3] allocation was worse (0.60629). Thus the mathematical mechanism is valid, but this experiment showed no downstream win.
- Agent confidence
- 9/10
- Baseline
- Uniform ranks [2,2], 32 coefficients, test MSE 0.5890389549554089.
- Idea
- Greedy spectral ranks [2,2], 32 coefficients, test MSE 0.5890389549554089; difference from uniform: 0.0. Bound residual/bound ratio: 0.4067372028; identity error: 2.65e-13.
Limitations: Only a small two-layer synthetic regression task was tested, not a transformer, language modeling, zeroth-order optimization, multiple budgets, or repeated random seeds. The greedy and uniform methods coincided at the single tested budget, so this does not establish that spectral allocation improves real fine-tuning efficiency.
How to run: python3 experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- experiment.py 4.9 KB View Raw
- report.md 1.4 KB View
- results.json 1.6 KB View Raw