ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
arXiv:2607.01125
2026
Optimization
2 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper provides a concrete way to make zeroth-order fine-tuning practical by replacing full weight perturbations with perturbations in activation-derived low-rank coordinates. The transferable asset is the exact factorization of a linear-layer gradient through the input activation SVD: only the right singular directions of the activation matrix can receive gradient signal, and rapidly decaying singular values provide an explicit approximation-error control. This yields a forward-only adapter whose zeroth-order dimension is reduced from the number of weight parameters to the number of retained activation directions times the output width, while leaving quantized base weights frozen. The most promising implementation is a per-layer activation-SVD ZO adapter combined with Adam-style updates and a rank-selection rule based on the measured singular-value tail.
Ideas from this paper
Unverified
Re-invented
2026
Freeze each pretrained linear weight matrix and optimize only a coefficient matrix in the top right-singular subspace of a one-shot activation SVD. Estimate the loss gradient with antithetic finite differences in coefficient space rather than in the full weight space, reducing both the number of forward perturbation directions and the variance of the estimator.
Useful8/10
Difficulty5/10
Novelty5/10
✗ Mechanism failed
2026
Use the activation singular-value spectrum to allocate a fixed zeroth-order parameter budget across layers instead of assigning the same rank everywhere. Layers with a large discarded singular-value tail receive more coefficient directions, while spectrally compressible layers use smaller adapters, preserving the gradient-relevant subspace under a global memory and query budget.
Useful7/10
Difficulty4/10
Novelty6/10