✓✓ Beats tuned baseline 2026

Directional Hölder Step Controller

Implementation & benchmark of arXiv:2607.22906 — Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity

Usefulness7/10

Difficulty5/10

Novelty6/10

Optimization Speedup Stability Accuracy Optimizer Training dynamics Optimization Approximation theory Dynamical systems

Source paper: Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity arXiv:2607.22906 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a fixed SGD learning rate with a per-update step selected from the positive curvature observed along the proposed direction. The controller estimates the directional Taylor remainder using one or two function evaluations, increases the step when the observed direction is benign, and backtracks only when the update fails a sufficient-decrease test.

Formulas

$$\left\|\nabla F(x)-\nabla F(y)\right\|\leq L\left\|x-y\right\|,\qquad x,y\in\mathbb{R}^{d}.$$

$$\min_{0\leq k<K}\left\|\nabla F(x_k)\right\|=O(K^{-1/2})$$

$$r_k=F(x_k+s_k)-F(x_k)-g_k^\top s_k,\qquad \widehat L_k=\frac{(1+\alpha)\max\{r_k,0\}}{\|s_k\|^{1+\alpha}+\varepsilon}.$$

$$s_k=-\eta_k g_k,\qquad \eta_k^\star=\left(\frac{\|g_k\|^{1-\alpha}}{\widehat L_k+\varepsilon}\right)^{1/\alpha},\qquad F(x_k+s_k)\leq F(x_k)-c\eta_k\|g_k\|^2.$$

Mathematical statement

The paper contrasts full gradient Lipschitz regularity, $\|\nabla F(x)-\nabla F(y)\|\le L\|x-y\|$, with a weaker one-sided condition that controls only the gradient variation entering the descent direction. The standard assumption is explicitly given as $\|\nabla F(x)-\nabla F(y)\|\leq L\left\|x-y\right\|$ for $x,y\in\mathbb{R}^d$, and under it gradient descent has the best-iterate guarantee $\min_{0\leq k<K}\|\nabla F(x_k)\|=O(K^{-1/2})$. For an implementable adaptation, let $g_k=\nabla F(x_k)$, $G_k=\|g_k\|$, proposed step $s_k=-\eta_k g_k$, and observed directional remainder $r_k=F(x_k+s_k)-F(x_k)-g_k^\top s_k$. Define the positive directional Hölder curvature estimate $\widehat L_k=(1+\alpha)\max(r_k,0)/(\|s_k\|^{1+\alpha}+\varepsilon)$, where $\alpha\in(0,1]$ is the Hölder exponent and $\varepsilon>0$ prevents division by zero. The one-sided descent model is $F(x_k+s)\le F(x_k)+g_k^\top s+\widehat L_k\|s\|^{1+\alpha}/(1+\alpha)$. Minimizing this model along $s=-\eta g_k$ gives the target scale $\eta_k^\star=(G_k^{1-\alpha}/(\widehat L_k+\varepsilon))^{1/\alpha}$. A safety factor $\rho\in(0,1)$ and the acceptance test $F(x_k+s_k)\le F(x_k)-c\eta_kG_k^2$, with $c\in(0,1)$, make the method robust to noisy estimates.

Implementation notes

Integrate this as an SGD learning-rate controller immediately before the parameter update, initially for full-batch or large-batch training where function values are sufficiently low variance. Maintain a scalar learning rate eta and compute the current minibatch gradient g and loss F0. Form a trial step s=-eta*g and evaluate the loss Ftrial at x+s on the same batch; do not yet commit the parameters. Compute the directional remainder r=Ftrial-F0-g dot s and the estimate Lhat=(1+alpha)*max(r,0)/(norm(s)^(1+alpha)+eps). Compute the model step eta_star=(norm(g)^(1-alpha)/(Lhat+eps))^(1/alpha), then clip it to [eta_min, eta_max] and multiply by a safety factor rho. In practice, use eta_candidate=clip(rho*eta_star, eta_min, eta_max), reevaluate the candidate loss, and accept if Fcandidate <= F0-c*eta_candidate*norm(g)^2. If rejected, repeatedly set eta_candidate=beta*eta_candidate with beta in [0.25,0.8] until acceptance or a maximum number of backtracks; if accepted, optionally set the next trial rate to min(eta_candidate/growth, eta_max) with growth around 0.9 to avoid oscillation. Pseudocode: compute g,F0; propose eta; evaluate Ftrial; r=Ftrial-F0+eta*norm(g)^2; Lhat=(1+alpha)*max(r,0)/(eta*norm(g))^(1+alpha); eta_new=clip(rho*(norm(g)^(1-alpha)/(Lhat+eps))^(1/alpha)); while F(x-eta_new*g)>F0-c*eta_new*norm(g)^2: eta_new*=beta; update x-=eta_new*g. The mathematical quantities are r, Lhat, the Hölder model, and eta_star; minibatch noise, alpha, rho, c, beta, and clipping limits must be selected empirically. First experiment: compare this controller against SGD with cosine decay, AdamW, and Armijo-style backtracking on a small ResNet-18/CIFAR-10 or two-layer MLP/MNIST using matched forward-backward FLOPs. Measure loss versus optimizer steps and wall-clock time, accepted-step fraction, learning-rate trajectory, gradient norm, and final accuracy. The expected signal is faster loss reduction at equal accepted updates, larger rates on regions where full gradient variation is mostly orthogonal to descent, and no increase in divergence or rejected-step frequency.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

How to read this: The implementation beat the baseline, but the claimed mechanism did not reproduce at benchmark scale. The gain may come from another effect; treat the cause as unresolved.

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified the alpha=1 Directional Hölder Step Controller. On quadratic sweeps, estimated directional curvature matched the exact value to approximately 1e-13 relative error, inverse-curvature learning-rate scaling matched eta·lambda=0.9, and the sufficient-decrease acceptance boundary matched the predicted 1.8 threshold. In the small fixed full-batch MLP test, the controller reached loss 0.1407 and 96.3% accuracy versus cosine SGD loss 0.6545 and 72.1%, with all steps accepted; this is promising but not a matched-FLOP or broad benchmark.

Agent confidence: 9/10

Baseline: Cosine SGD: final loss 0.6545, accuracy 72.1%, loss at steps 10/50 = 0.6695/0.6557, mean learning rate 0.0758, wall time 0.325 s.
Idea: Directional controller: final loss 0.1407, accuracy 96.3%, loss at steps 10/50 = 0.6487/0.2783, accepted fraction 100%, mean learning rate 1.6610, wall time 0.312 s. Toy curvature relative error ranged from 4.1e-13 to 3.4e-16; observed eta·lambda was 0.9000000000 across lambda 0.1–10; boundary predictions matched all 6 tests.

Limitations: Only alpha=1 was tested, using exact full-batch losses on a tiny 2x16 MLP and synthetic data. AdamW, minibatch noise, larger models/datasets, rigorous matched-FLOP accounting, repeated seeds, and wall-clock scaling were not tested.

How to run: python3 directional_controller_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.22906) Sep 2, 2026

Beats the tuned baseline by 45.9% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 16.6689
Idea mean: 9.0252
Effect (Δ): -7.6436 (−45.9%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±17.6%
Mechanism: Not confirmed ✗
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the small tabular track was tested: 400 training examples, 200 test examples, 8 epochs, and batch size 128. Other tracks, larger datasets, FLOP-normalized or wall-clock speed, alpha below 1, and equal-forward-evaluation accounting were not tested. The controller uses extra forward evaluations, and its mechanism signature failed confirmation.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Worked ✓

Agent confidence: 8/10

The Directional Hölder controller was implemented as a custom optimizer loop on the registered tabular/Friedman regression track with the shared mlp_tiny architecture. It significantly beat tuned Adam on all 8 paired seeds (delta_mean=-7.643633, p_value=0.0081), satisfying the benchmark criterion for worked=true. The trained-model mechanism signature was not confirmed: mean observed-to-predicted step ratio was 5.0215.

Baseline: Tuned Adam lr=0.01: mean test MSE 16.668858, std 2.727546.
Idea: Directional controller initial lr=0.001: mean test MSE 9.025225, std 1.738034.

Benchmark result