arXiv:2607.03402 2026 Optimization 1 ideas extracted · analyzed Aug 30, 2026
The paper transfers a useful randomized linear-algebra pattern to neural-network training: maintain low-rank factors while replacing expensive global QR orthogonalization by sketch-based orthogonalization. The important asset is not merely low-rank compression, which is already common, but the distinction between an oblique sketched projection and an approximately orthogonal projection; the former can fail when the update has a large component outside the current low-rank tangent space. A practical target is distributed training of low-rank adapters or optimizer states, where randomized Cholesky QR can reduce synchronization and convert basis maintenance from BLAS-2 to BLAS-3 operations. The first experiment should test whether this preserves loss and final accuracy while reducing communication and orthogonalization cost.
Apply the paper's orthogonal-sketch idea to a trainable low-rank weight update or low-rank optimizer state. After every optimizer step, use a row sketch and Cholesky QR to recondition the factor basis, then compute the update with the standard tangent-space projector rather than an oblique sketched projector.