Adaptive Hybrid Subspace Levenberg Marquardt Algorithm with Adequacy Monitor for Large Scale Least Squares Problems
arXiv:2608.25524
2026
Optimization
2 ideas extracted · analyzed Aug 29, 2026
What the math gives to ML
The paper offers a practical large-scale second-order optimizer built around an adaptively enriched low-dimensional parameter subspace rather than a full Gauss–Newton solve. Its transferable asset is the explicit combination of descent directions, accepted-step memory, Krylov curvature probes, and randomized curvature directions, together with a projected-gradient adequacy test that detects when the subspace is missing important optimization information. A second useful mechanism separates step acceptance from damping adaptation: Armijo backtracking chooses the step length, while the actual-to-predicted reduction ratio changes the LM damping without repeatedly solving new damped systems. The most direct neural-network test is a minibatch HSLM optimizer on small and medium regression models, measuring loss decrease per wall-clock time and per Jacobian-vector product against AdamW, full LM, and Krylov LM.
Ideas from this paper
✗ Mechanism failed
2026
Replace a full neural-network Gauss–Newton solve with a damped solve in an adaptively constructed low-dimensional parameter subspace. The subspace contains the current gradient, recent accepted updates, Krylov curvature directions, and randomized Jacobian-curvature probes, and is enlarged whenever its projected gradient fails to capture enough descent information.
Useful8/10
Difficulty6/10
Novelty7/10
✓ Mechanism works
Re-invented
2026
Use one projected LM direction and let Armijo backtracking decide the accepted step length, instead of repeatedly resolving the damped system whenever a trial step is rejected. Independently update the damping parameter from the ratio of actual to predicted reduction, preserving the trust-region-like adaptivity of LM while reducing expensive curvature solves.
Useful7/10
Difficulty4/10
Novelty8/10