From Manifold Identification to Newton Acceleration on Intersections: Sparse Stiefel Optimization

What the math gives to ML

The paper develops a safeguarded second-order method for optimization problems combining nonsmooth sparsity with an orthogonality constraint. Its transferable asset is finite identification of an active support followed by Newton-CG steps on the resulting smooth intersection, together with damping rules that fall back to proximal manifold-gradient steps when projection or Newton steps fail. For neural networks, this suggests training sparse orthogonal linear operators—such as attention projections, SSM transition matrices, or compressed MLP layers—with an optimizer that first discovers a support pattern and then exploits the lower-dimensional smooth manifold instead of continuing to pay for generic nonsmooth updates. The off-diagonal constraint perturbation is an additional mechanism for handling sparse patterns whose active manifold intersects the Stiefel manifold nontransversely.