Uniform-History Reset Optimizer

Integrate the mechanism into SGD or Adam after the ordinary gradient update and before the next minibatch. Maintain a parameter snapshot buffer containing every \(M\)-th iterate, or use reservoir sampling when storing all iterates is too expensive. At step \(k\), compute the normal update \(\theta'_{k+1}=\theta_k-\eta g_k\). Draw \(u\sim U[0,1]\); if \(u<r\Delta t\), choose an index \(J\) uniformly from the buffer and set \(\theta_{k+1}=\theta_J\); otherwise set \(\theta_{k+1}=\theta'_{k+1}\). Reset optimizer momentum together with the parameters, or test both momentum-preserving and momentum-resetting variants. A deterministic low-variance variant maintains an online historical mean \(\bar\theta_k\) and applies \(\theta_{k+1}=\theta'_{k+1}+\eta r(\bar\theta_k-\theta'_{k+1})\). The paper supplies the uniform-history kernel and its non-Markovian averaging mechanism; the gradient update, discrete-time implementation, buffer thinning, and optimizer-state policy are adaptations. First use the quadratic loss \(L(x,y)=\frac12(\lambda x^2-a y^2)\), with \(a>0\), and compare ordinary SGD with reset rates \(r/a\in\{0.5,0.8,1.0,1.2,2.0\}\). Measure the long-time growth of \(|y_k|\) and fit the stability boundary; the prediction is a transition near \(r_c\approx a\), with measured \(r_c/a\) within 20% after accounting for discrete step size. Then train a two-layer MLP on MNIST using SGD, testing several reset rates and a buffer of at least 100 snapshots. The predicted signatures are reduced parameter-norm divergence above the quadratic threshold, slower approximately algebraic relaxation rather than a single exponential, and measurable returns to earlier parameter basins. Compare accuracy, loss variance, reset frequency, and sensitivity to buffer size.