Double-Inertial Halpern Optimizer

(1) Integrate at the parameter-update stage, after computing a minibatch gradient and before writing the optimizer state. Maintain three parameter vectors: current θk, previous θk−1, and second-previous θk−2. Use r=0 initially, so the proximal map is the identity; optionally use weight decay or a group sparsity penalty as r. Compute the extrapolated point yk=θk+βk(θk−θk−1)+δk(θk−1−θk−2). Evaluate the minibatch gradient at yk, form zk=proxγr(yk−γgk), and update θk+1=(1−αk)θ0+αkzk. (2) Pseudocode: initialize θ−1=θ−2=θ0; for each minibatch, set d1=θk−θk−1, d2=θk−1−θk−2, y=θk+βkd1+δkd2, g=∇θℓ(y;B), z=proxγr(y−γg), then θk+1=(1−αk)θ0+αkz and shift the history. Start conservatively with αk=min(1,(k+1)/(k+10)), βk=0.5√(k/(k+1)), and δk=0.1√(k/(k+2)); tune these schedules against AdamW. (3) The paper supplies the operator structure and asymptotic residual and velocity targets; the implementation estimates the stochastic gradient empirically. Log the normalized velocity ||θk−θk−1||/||θk|| and the computable forward-backward residual ||θk−zk||, using exponential moving averages. If loss spikes, reset βk=δk=0 for 100 steps and multiply both by 0.9 thereafter. (4) First experiment: train a 2-layer MLP and a small ResNet-18 on MNIST or CIFAR-10, comparing AdamW, SGD with momentum, and the proposed optimizer at matched minibatches, parameter updates, and wall-clock time. Measure training loss versus updates, validation accuracy, gradient-norm variance, and number of restarts. The desired signal is faster loss reduction at equal FLOPs, lower late-training velocity or residual, or equal accuracy with fewer optimizer steps; failure is persistent overshoot or no improvement over tuned AdamW.