No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

What the math gives to ML

The paper identifies low-rank gradient subspaces as noisy coordinate systems rather than slowly evolving objects, so attempting to track individual basis directions is fundamentally unreliable. Its most transferable result is an optimizer-state rule: the first moment can be transported through the overlap between old and new bases, whereas the coordinate-wise second moment should not be rotated blindly because squaring is not rotation-equivariant. A practical adaptation is to transport momentum with the basis-overlap matrix and replace stale anisotropic second moments by a rotation-blind isotropic statistic, optionally shortening its decay because refreshes invalidate long-memory coordinate statistics. This directly targets low-rank optimizers such as GaLore and is testable without changing the model architecture.