Directional Hölder Step Controller

Implementation

directional_controller_experiment.py implements the alpha=1 controller for a differentiable parameter vector. It probes the maintained rate, computes

r = F(theta - eta*g) - F(theta) + eta*||g||^2
Lhat = 2*max(r, 0)/(eta*||g||)^2
eta_star = 1/Lhat
eta_candidate = clip(rho*eta_star)

and applies sufficient-decrease backtracking. The experiment also compares it with cosine-decay SGD on a fixed full-batch nonlinear binary classification problem.

Toy mechanism verification

For a quadratic F(theta)=theta^T H theta/2, alpha=1 predicts:

Lhat exactly equals directional curvature g^T H g / ||g||^2, independently of probe step.
With safety factor rho=0.9, accepted eta*lambda equals 0.9 for H=lambda I, hence eta scales as 1/lambda.
The sufficient-decrease test accepts exactly when eta*lambda <= 2(1-c); with c=0.1, the boundary is 1.8.

Observed results are in results.json:

Curvature relative error across lambda scales 0.1 to 10: 4.1e-13 down to 3.4e-16.
Inverse scaling: observed eta*lambda is 0.9000000000 for every tested lambda (0.1, 0.3, 1, 3, 10).
Boundary tests at eta*lambda={1.7,1.9,2.1} exactly match predicted accept/reject outcomes for lambda 1 and 3.

Thus the proposed mechanism manifests in the controlled setting.

Mini-experiment

Fixed seed 1296, 512 examples, 2x16 tanh MLP, 100 full-batch updates on CUDA:

| metric | cosine SGD | directional controller | |---|---:|---:| | final loss | 0.6545 | 0.1407 | | loss at step 10 | 0.6695 | 0.6487 | | loss at step 50 | 0.6557 | 0.2783 | | accuracy | 72.1% | 96.3% | | accepted fraction | 100% | 100% | | mean learning rate | 0.0758 | 1.6610 | | wall time (s) | 0.325 | 0.312 |

This is a promising signal, but not a matched-FLOP benchmark: the controller evaluates additional trial/candidate losses, while GPU timing at this tiny scale is noisy. No AdamW, minibatch-noise, alpha<1, or larger vision benchmark was tested.

Reproduce

/home/maxwelhelp/main/bin/python3 directional_controller_experiment.py