Renewal-reset optimizer

Integrate the method at the optimizer/checkpoint level, not inside individual layers. Maintain trainable parameters $\theta$, a reference checkpoint $\theta_{\rm ref}$, and its best observed loss $L_{\rm ref}$. At the beginning of each renewal episode, set $\theta\leftarrow\theta_{\rm ref}$ and reset Adam's first and second moments to zero. Sample an episode length $R$ from a chosen discrete distribution, initially a truncated geometric law $r_j=h(1-h)^{j-1}$ for $1\le j<K$ with residual mass at $K$, or a two-point mixture such as $R\in\{K/4,K\}$. Then run ordinary minibatch AdamW for $R$ steps. After each episode, update $\theta_{\rm ref}$ to the lowest-loss parameters seen so far, optionally using an EMA checkpoint. Pseudocode: for episode: theta=theta_ref; m=v=0; sample R~r; for k in 1..R: g=grad(L(theta)); m=beta1*m+(1-beta1)*g; v=beta2*v+(1-beta2)*g^2; theta=AdamW(theta,m,v); record loss; if loss<best: theta_ref=theta; best=loss. Estimate $q_k$ empirically from multiple short uninterrupted runs and estimate $P_k$ from reset runs; test the discrete renewal recursion using histogrammed loss or low-dimensional PCA coordinates rather than the full parameter density. The first cheap experiment should use a two-layer MLP on MNIST or a small CNN on CIFAR-10, comparing ordinary AdamW, fixed-period checkpoint restart, and geometric renewal reset at equal optimizer steps and equal minibatch samples. Success means lower median steps to a target validation loss, fewer catastrophic high-loss runs, or improved final accuracy; failure is increased variance or no improvement over fixed restarts.