Time-Delay Error Adaptive Optimizer

1. Integration point: implement this as a wrapper around SGD or momentum SGD, initially avoiding Adam's internal normalization so that the meaning of the interval is clear. Interpret $h_k$ as the multiplier of the proposed parameter displacement: compute a direction $d_k$ and update $w_{k+1}=w_k-h_kd_k$. Store the last three gradients, their intervals, and the last residual. A fixed probe minibatch can be used to reduce measurement noise while training gradients remain stochastic. 2. Pseudocode: compute $g_k$ and the base optimizer direction $d_k$; form $\widehat g_k=g_{k-1}+(h_k/h_{k-1})(g_{k-1}-g_{k-2})$; calculate $r_k=g_k-\widehat g_k$; compute $h_{\mathrm{raw}}=\sqrt{\varepsilon/(\|r_k\|_2+\delta)}$; project it onto $[h_{\min},h_{\max}]$ and then onto $[h_k/q,qh_k]$; apply $w\leftarrow w-h_kd_k$. For noisy minibatches, use $R_k=\beta R_{k-1}+(1-\beta)\|r_k\|_2$ and substitute $R_k$ for the raw norm. Initialize with three fixed-step updates. 3. Taken from the paper: quadratic error scaling, the algebraic square-root scheduler, and bounded step-ratio design. Estimated empirically: $C$, residual noise, and the useful tolerance $\varepsilon$. 4. First experiment: train a two-layer MLP on MNIST with full-batch or large-batch SGD, comparing fixed-step SGD, cosine scheduling, and this scheduler at matched average step size. Sweep fixed $h$ values and measure residual norm versus $h$; fit a log-log slope. The quantitative prediction is a slope of approximately $2\pm0.3$ in the smooth full-batch regime, a stable operating interval near $h^*=\sqrt{\varepsilon/C}$, and bounded ratios $h_{k+1}/h_k\in[1/q,q]$. Test whether the residual settles within 20 percent of $\varepsilon$ and whether adaptive shortening precedes loss spikes when the fixed-step baseline crosses its divergence boundary.