Statistical Inference for Rank Allocation in Low-Rank Adaptation

What the math gives to ML

The paper turns LoRA rank allocation into a one-sided statistical decision problem rather than a deterministic gradient-importance heuristic. Its transferable asset is the separation between a latent component contribution and a noisy estimate generated by minibatch sampling, momentum, and adaptive optimizers, together with uncertainty quantification. A practical adaptation is to collect per-step contribution samples for every LoRA rank-one component, estimate a noise-aware standard error, and prune components whose evidence for useful contribution is weakest. This can make rank allocation less sensitive to transient gradients while providing an interpretable compute-versus-accuracy control knob.