The Loss Does Not See the Basis, but Adam Does
arXiv:2608.05136
2026
Optimization
2 ideas extracted · analyzed Aug 31, 2026
What the math gives to ML
The paper identifies gauge equivariance as the mathematical condition separating optimizers that preserve the implicit low-rank behavior of factored models from coordinate-wise methods that destroy it. For a factorization W=UV^\top, the loss is invariant under simultaneous latent-basis rotations (U,V)\mapsto(UQ,VQ), but Adam's coordinate-wise normalization is not, so equivalent parameterizations follow different optimization paths. The transferable construction is to replace coordinate-wise preconditioning with a shared scalar or Gram-matrix-based left preconditioner, then optionally control its magnitude with a gauge-invariant spectral schedule.
Ideas from this paper
△ Mechanism confirmed, baseline not beaten
2026
Replace Adam's elementwise denominator on factor matrices with a gauge-equivariant denominator shared across latent coordinates, or with a left preconditioner constructed from Gram matrices. The resulting update treats all equivalent bases of the latent space identically, allowing the low-rank implicit bias of factorized gradient flow to survive adaptive optimization.
Useful8/10
Difficulty4/10
Novelty7/10
✗ Failed on benchmark
2026
Use the evolving singular spectrum of the represented matrix W_t=U_tV_t^{\top} to modulate one common, gauge-equivariant learning rate. Slow the shared update when spectral mass accumulates outside the intended low-rank subspace, preventing adaptive dynamics from amplifying nuisance tail directions while retaining the shared-rate structure needed for low-rank recovery.
Useful7/10
Difficulty5/10
Novelty8/10