Adaptive, Matrix-Free Low-Rank Approximation

What the math gives to ML

This paper provides a constructive way to choose the rank of a low-rank approximation from an explicit error tolerance rather than fixing the rank in advance. Its transferable assets are matrix-free randomized residual estimation, large block construction followed by rank pruning, and a forward-only variant that avoids adjoint products. These mechanisms can turn low-rank neural layers and attention projections into tolerance-controlled modules whose rank adapts per layer and per checkpoint. The most direct experiment is to replace dense linear layers with adaptively built low-rank factors and test whether the same validation loss is achieved with fewer parameters, matvecs, or inference FLOPs.