# Directional Hölder Step Controller

- ID: 1296
- Canonical URL: https://synthcore.org/idea/1296/directional-h-lder-step-controller
- API JSON: https://synthcore.org/api/idea/1296.json
- API Markdown: https://synthcore.org/api/idea/1296.md
- Verification status: beats_baseline
- Source: [arXiv:2607.22906](https://arxiv.org/abs/2607.22906)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, training-dynamics
- Math tags: optimization, approximation-theory, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace a fixed SGD learning rate with a per-update step selected from the positive curvature observed along the proposed direction. The controller estimates the directional Taylor remainder using one or two function evaluations, increases the step when the observed direction is benign, and backtracks only when the update fails a sufficient-decrease test.

## Mathematical statement

The paper contrasts full gradient Lipschitz regularity, \(\|\nabla F(x)-\nabla F(y)\|\le L\|x-y\|\), with a weaker one-sided condition that controls only the gradient variation entering the descent direction. The standard assumption is explicitly given as \(\|\nabla F(x)-\nabla F(y)\|\leq L\left\|x-y\right\|\) for \(x,y\in\mathbb{R}^d\), and under it gradient descent has the best-iterate guarantee \(\min_{0\leq k<K}\|\nabla F(x_k)\|=O(K^{-1/2})\). For an implementable adaptation, let \(g_k=\nabla F(x_k)\), \(G_k=\|g_k\|\), proposed step \(s_k=-\eta_k g_k\), and observed directional remainder \(r_k=F(x_k+s_k)-F(x_k)-g_k^\top s_k\). Define the positive directional Hölder curvature estimate \(\widehat L_k=(1+\alpha)\max(r_k,0)/(\|s_k\|^{1+\alpha}+\varepsilon)\), where \(\alpha\in(0,1]\) is the Hölder exponent and \(\varepsilon>0\) prevents division by zero. The one-sided descent model is \(F(x_k+s)\le F(x_k)+g_k^\top s+\widehat L_k\|s\|^{1+\alpha}/(1+\alpha)\). Minimizing this model along \(s=-\eta g_k\) gives the target scale \(\eta_k^\star=(G_k^{1-\alpha}/(\widehat L_k+\varepsilon))^{1/\alpha}\). A safety factor \(\rho\in(0,1)\) and the acceptance test \(F(x_k+s_k)\le F(x_k)-c\eta_kG_k^2\), with \(c\in(0,1)\), make the method robust to noisy estimates.

## Key formulas

- $$\left\|\nabla F(x)-\nabla F(y)\right\|\leq L\left\|x-y\right\|,\qquad x,y\in\mathbb{R}^{d}.$$
- $$\min_{0\leq k<K}\left\|\nabla F(x_k)\right\|=O(K^{-1/2})$$
- $$r_k=F(x_k+s_k)-F(x_k)-g_k^\top s_k,\qquad \widehat L_k=\frac{(1+\alpha)\max\{r_k,0\}}{\|s_k\|^{1+\alpha}+\varepsilon}.$$
- $$s_k=-\eta_k g_k,\qquad \eta_k^\star=\left(\frac{\|g_k\|^{1-\alpha}}{\widehat L_k+\varepsilon}\right)^{1/\alpha},\qquad F(x_k+s_k)\leq F(x_k)-c\eta_k\|g_k\|^2.$$

## Implementation notes

Integrate this as an SGD learning-rate controller immediately before the parameter update, initially for full-batch or large-batch training where function values are sufficiently low variance. Maintain a scalar learning rate eta and compute the current minibatch gradient g and loss F0. Form a trial step s=-eta*g and evaluate the loss Ftrial at x+s on the same batch; do not yet commit the parameters. Compute the directional remainder r=Ftrial-F0-g dot s and the estimate Lhat=(1+alpha)*max(r,0)/(norm(s)^(1+alpha)+eps). Compute the model step eta_star=(norm(g)^(1-alpha)/(Lhat+eps))^(1/alpha), then clip it to [eta_min, eta_max] and multiply by a safety factor rho. In practice, use eta_candidate=clip(rho*eta_star, eta_min, eta_max), reevaluate the candidate loss, and accept if Fcandidate <= F0-c*eta_candidate*norm(g)^2. If rejected, repeatedly set eta_candidate=beta*eta_candidate with beta in [0.25,0.8] until acceptance or a maximum number of backtracks; if accepted, optionally set the next trial rate to min(eta_candidate/growth, eta_max) with growth around 0.9 to avoid oscillation. Pseudocode: compute g,F0; propose eta; evaluate Ftrial; r=Ftrial-F0+eta*norm(g)^2; Lhat=(1+alpha)*max(r,0)/(eta*norm(g))^(1+alpha); eta_new=clip(rho*(norm(g)^(1-alpha)/(Lhat+eps))^(1/alpha)); while F(x-eta_new*g)>F0-c*eta_new*norm(g)^2: eta_new*=beta; update x-=eta_new*g. The mathematical quantities are r, Lhat, the Hölder model, and eta_star; minibatch noise, alpha, rho, c, beta, and clipping limits must be selected empirically. First experiment: compare this controller against SGD with cosine decay, AdamW, and Armijo-style backtracking on a small ResNet-18/CIFAR-10 or two-layer MLP/MNIST using matched forward-backward FLOPs. Measure loss versus optimizer steps and wall-clock time, accepted-step fraction, learning-rate trajectory, gradient norm, and final accuracy. The expected signal is faster loss reduction at equal accepted updates, larger rates on regions where full gradient variation is mostly orthogonal to descent, and no increase in divergence or rejected-step frequency.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built and numerically verified the alpha=1 Directional Hölder Step Controller. On quadratic sweeps, estimated directional curvature matched the exact value to approximately 1e-13 relative error, inverse-curvature learning-rate scaling matched eta·lambda=0.9, and the sufficient-decrease acceptance boundary matched the predicted 1.8 threshold. In the small fixed full-batch MLP test, the controller reached loss 0.1407 and 96.3% accuracy versus cosine SGD loss 0.6545 and 72.1%, with all steps accepted; this is promising but not a matched-FLOP or broad benchmark.

### Mechanism check

- Verdict: Built and numerically verified the alpha=1 Directional Hölder Step Controller. On quadratic sweeps, estimated directional curvature matched the exact value to approximately 1e-13 relative error, inverse-curvature learning-rate scaling matched eta·lambda=0.9, and the sufficient-decrease acceptance boundary matched the predicted 1.8 threshold. In the small fixed full-batch MLP test, the controller reached loss 0.1407 and 96.3% accuracy versus cosine SGD loss 0.6545 and 72.1%, with all steps accepted; this is promising but not a matched-FLOP or broad benchmark.
- Confidence: 9/10
- Limitations: Only alpha=1 was tested, using exact full-batch losses on a tiny 2x16 MLP and synthetic data. AdamW, minibatch noise, larger models/datasets, rigorous matched-FLOP accounting, repeated seeds, and wall-clock scaling were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [REPORT.md](https://synthcore.org/code/514/REPORT.md)
- [bench_report.json](https://synthcore.org/code/514/bench_report.json)
- [bench_run.py](https://synthcore.org/code/514/bench_run.py)
- [directional_controller_experiment.py](https://synthcore.org/code/514/directional_controller_experiment.py)
- [report.md](https://synthcore.org/code/514/report.md)
- [report_bench_2026-08-31T163240.md](https://synthcore.org/code/514/report_bench_2026-08-31T163240.md)
- [results.json](https://synthcore.org/code/514/results.json)
- [Download all files as ZIP](https://synthcore.org/download/514)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.