A Gaussian smoothing-based zeroth-order method for Goldstein second-order stationarity
arXiv:2607.21258
2026
Optimization
1 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper replaces classical Hessian stationarity for continuously differentiable objectives with locally Lipschitz gradients by a Goldstein second-order notion that remains meaningful when the Hessian is discontinuous or undefined at many points. Its transferable asset is a derivative-free cubic-regularization step: Gaussian-smoothed function queries estimate both a gradient direction and a curvature model, while homotopy in the smoothing radius progressively resolves sharper structure. This suggests a zeroth-order optimizer for black-box neural-network fine-tuning that can detect negative curvature and escape approximate saddles without backpropagation.
Ideas from this paper
Unverified
2026
Replace occasional gradient updates in a neural-network optimizer with a Gaussian-smoothed cubic-regularization step estimated entirely from loss evaluations. The cubic term stabilizes steps under uncertain curvature, while the Goldstein neighborhood interpretation makes the method robust to discontinuous or rapidly varying Hessians and gives it an explicit negative-curvature escape mechanism.
Useful6/10
Difficulty6/10
Novelty6/10