Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers
arXiv:2607.14516
2026
Optimization
2 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper identifies a concrete failure mode in adaptive higher-order optimizers: an error controller can be present but operationally inert when the step-size growth cap is reached immediately and rejected steps are never actually rejected. Its transferable asset is an embedded Runge–Kutta error estimate coupled to a genuine accept/reject controller whose error is measured on the optimizer map actually applied to parameters. A second, cheaper asset is averaging gradients evaluated along a short predicted trajectory, which may provide implicit regularization without the full 3–4x cost of RK stages. Both mechanisms should be evaluated under strict backward-pass matching against Adam, AdamW, RMSprop, and NAdam.
Ideas from this paper
Unverified
2026
Replace a fixed Adam update by an embedded Bogacki–Shampine RK3(2) proposal with a genuine accept/reject controller. Measure error between the two actual Adam parameter maps, rather than only between raw gradient estimates, and charge every gradient evaluation against the training compute budget.
Useful6/10
Difficulty6/10
Novelty6/10
Unverified
2026
Use a two-gradient predictor-corrector average as the gradient supplied to Adam, retaining trajectory smoothing while avoiding the three or four gradient evaluations required by full RK3. Vary the mixing coefficient to test whether the reported regularization comes from gradient averaging itself rather than from high-order integration.
Useful5/10
Difficulty4/10
Novelty5/10