Unverified 2026

Compute-Matched Embedded RK Adam

Implementation & benchmark of arXiv:2607.14516 — Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

Usefulness6/10

Difficulty6/10

Novelty6/10

Optimization Stability Accuracy Generalization Optimizer Training dynamics Numerical analysis Optimization Dynamical systems

Source paper: Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers arXiv:2607.14516 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a fixed Adam update by an embedded Bogacki–Shampine RK3(2) proposal with a genuine accept/reject controller. Measure error between the two actual Adam parameter maps, rather than only between raw gradient estimates, and charge every gradient evaluation against the training compute budget.

Formulas

$$g_{3}=\tfrac{2}{9}k_{1}+\tfrac{1}{3}k_{2}+\tfrac{4}{9}k_{3}$$

$$k_1=\nabla L(\theta),\quad k_2=\nabla L(\theta-\tfrac{h}{2}k_1),\quad k_3=\nabla L(\theta-\tfrac{3h}{4}k_2),\quad \theta_3=\theta-hg_3$$

$$g_{2}=\tfrac{7}{24}k_{1}+\tfrac{1}{4}k_{2}+\tfrac{1}{3}k_{3}+\tfrac{1}{8}k_{4},\quad k_4=\nabla L(\theta_3),\quad e=h(g_3-g_2)$$

$$\phi=\min\!\big(\max(0.9\,\widehat{\mathrm{err}}^{-1/3},\,0.5),\,2.0\big),\qquad h\leftarrow\mathrm{clip}(h\phi,[0.1\,\mathrm{lr},h_{\max}])$$

Mathematical statement

For gradient flow $\dot\theta=-\nabla L(\theta)$, let $k_i$ be gradients evaluated at intermediate parameters and $h$ the trial step size. The third-order embedded gradient is $g_3=\frac{2}{9}k_1+\frac{1}{3}k_2+\frac{4}{9}k_3$. Define $k_1=\nabla L(\theta)$, $k_2=\nabla L(\theta-\frac{h}{2}k_1)$, $k_3=\nabla L(\theta-\frac{3h}{4}k_2)$, $\theta_3=\theta-hg_3$, and $k_4=\nabla L(\theta_3)$. The second-order estimate is $g_2=\frac{7}{24}k_1+\frac{1}{4}k_2+\frac{1}{3}k_3+\frac{1}{8}k_4$, with raw discrepancy $e=h(g_3-g_2)$. Let $A_h(\theta,m,v;g)$ denote one Adam update from parameters $\theta$ and moments $m,v$ using gradient $g$. Estimate applied-map error as $\widehat{\mathrm{err}}=\|A_h(\theta,m,v;g_3)-A_h(\theta,m,v;g_2)\|_2/(\|\theta\|_2+\epsilon)$. Use $\phi=\min(\max(0.9\widehat{\mathrm{err}}^{-1/3},0.5),2.0)$ and $h\leftarrow\mathrm{clip}(h\phi,[0.1\,\mathrm{lr},h_{\max}])$. A trial with error above tolerance is rejected and recomputed.

Implementation notes

(1) Integrate this into the optimizer step immediately before Adam's moment update. Save the current parameters $\theta$, first moment $m$, and second moment $v$; only commit the accepted candidate state. Use a gradient-evaluation counter so all methods are compared at equal backward-pass cost. For one minibatch, compute $k_1$ at $\theta$, $k_2$ at $\theta-hk_1/2$, $k_3$ at $\theta-3hk_2/4$, form $g_3$, compute $k_4$ at $\theta-hg_3$, and form $g_2$. (2) Pseudocode: save state; evaluate the four stages; construct $S_3=A_h(\theta,m,v;g_3)$ and $S_2=A_h(\theta,m,v;g_2)$; compute the normalized distance between their parameter outputs; if error is below tolerance, commit $S_3$, update $h$ using $\phi$, and optionally reuse $k_4$ through FSAL; otherwise commit nothing, shrink $h$, and repeat the same minibatch until acceptance or a retry limit. (3) Compute RK coefficients exactly, estimate error from the two Adam maps, and log rejection rate, normalized error, step size, gradient evaluations per accepted step, and whether $h$ is pinned at its cap. (4) First test CIFAR-10 with a small ResNet-18 under a fixed 10,000-backward-pass budget. Compare Adam, the paper's no-rejection controller, and the corrected controller over matched learning-rate and tolerance sweeps. Success requires lower validation loss or higher test accuracy at equal backward evaluations, plus a nonzero but not excessive rejection rate and a step size that is not permanently cap-pinned.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.