Late-Time Fractional-Order Optimizer

(1) Integration point: add this as an AdamW replacement or wrapper around the parameter update. Maintain a short history of the scalar training signal $M_k$, preferably an exponential moving average of the minibatch loss above a positive floor, $M_k=\operatorname{EMA}(\ell_k-\ell_{\min}+\epsilon)$. Every $K$ optimizer steps compare the current signal with the signal approximately $K(\rho-1)$ steps earlier; use $m=1$, $\rho\in[1.5,2]$, and clip $\alpha$ to $[0.2,0.95]$. (2) Pseudocode: initialize $\theta$, gradient history $g_0=0$, a signal history, and $\alpha=0.5$. At step $k$, compute minibatch gradient $g_k$, update $M_k=\beta_M M_{k-1}+(1-\beta_M)(\ell_k-\ell_{\min}+\epsilon)$, and append it to a ring buffer. If $k\geq k_{\rm warmup}$ and the older value $M_{k-L}$ exists, calculate $r=M_k/M_{k-L}$, reject the estimate if $r\notin(0,1)$ or if the last $q$ ratios are not monotone, otherwise set $\alpha\leftarrow\operatorname{clip}[-\log r/\log\rho]$ with $\rho=k/(k-L)$ and exponentially smooth the estimate. Implement the fractional update using stored gradients: $v_k=\Gamma(\alpha)^{-1}\sum_{j=0}^{H-1}w_{k,j}(\alpha)g_{k-j}$, with piecewise-constant interval weights $w_{k,j}=((j+1)\Delta t)^\alpha-(j\Delta t)^\alpha$. Update $\theta_{k+1}=\theta_k-\eta v_k$ and optionally apply AdamW diagonal normalization to each stored gradient before accumulation. (3) Computed from the paper: the power-law ratio estimator and Caputo kernel exponent. Estimated empirically: the late-time window, loss floor, $m$, smoothing constants, and validity of monotonicity. (4) First experiment: train a small Transformer or ResNet on CIFAR-10 with equal token or update budgets, comparing AdamW, fixed-order fractional memory for $\alpha\in\{0.3,0.5,0.7,0.9\}$, and the adaptive method. Use history lengths $H\in\{32,128\}$ and measure validation accuracy, loss at equal FLOPs, gradient-norm variance, and sensitivity across random seeds. The method succeeds if it improves late-plateau loss or accuracy without destabilizing early training; useful signals are faster loss descent after warmup and lower variance than the best fixed-order baseline.