Unverified 2026

Square-Summable Noise Guard

Implementation & benchmark of arXiv:2608.19074 — A Mini-Batch Counterexample to Last-Iterate Convergence in Definable Optimization

Usefulness5/10

Difficulty3/10

Novelty3/10

Training Stability Generalization Optimizer Training dynamics Loss Stochastic processes Probability Convex analysis Dynamical systems

Source paper: A Mini-Batch Counterexample to Last-Iterate Convergence in Definable Optimization arXiv:2608.19074 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a late-training safeguard that decays the effective stochastic update scale fast enough to make the accumulated update variance finite. The safeguard is motivated by the paper's bounded reflected-random-walk counterexample: iterates can keep traversing an entire flat critical set forever even though the stepsize tends to zero and the objective values remain optimal.

Formulas

$$J(w)=\frac{1}{n}\sum_{i=1}^{n}f_i(w),\qquad w_{k+1}=w_k-\alpha_k d_k,\qquad d_k\in\frac{1}{|B_k|}\sum_{i\in B_k}D_i(w_k).$$

$$\alpha_k=o(1/\log k),\qquad \sum_{k=0}^{\infty}\alpha_k^2=\infty.$$

$$D_J(x)=\partial_C\phi(x)=\begin{cases}\{-1\},&x<-1,\\[-1,0],&x=-1,\\\{0\},&-1<x<1,\\[0,1],&x=1,\\\{1\},&x>1,\end{cases}\qquad \operatorname{crit}_J=[-1,1],\quad J|_{[-1,1]}\equiv0.$$

$$\eta_k=\eta_0(k+1)^{-p},\qquad \frac12<p\le1,\qquad \sum_k\eta_k=\infty,\quad \sum_k\eta_k^2<\infty,\qquad w_{k+1}=w_k-\eta_k d_k.$$

Mathematical statement

The paper studies the finite-sum objective $J(w)=\frac{1}{n}\sum_{i=1}^n f_i(w)$ and the mini-batch stochastic approximation update $w_{k+1}=w_k-\alpha_k d_k$, where $B_k$ is an independently sampled nonempty mini-batch and $d_k\in |B_k|^{-1}\sum_{i\in B_k}D_i(w_k)$. Its counterexample uses $\alpha_k=o(1/\log k)$, yet the iterates do not converge. In the explicit two-summand construction, the averaged field is $D_J(x)=\partial_C\phi(x)$, with $D_J(x)=\{0\}$ for $-1<x<1$, and the critical set is $\operatorname{crit}_J=[-1,1]$ with $J|_{[-1,1]}\equiv0$. For each batch $B\in\mathcal B=\{\{1\},\{2\},\{1,2\}\}$, the selected direction is the minimum-norm element $g_B(x)=\operatorname{proj}_{G_B(x)}(0)$, where $G_B(x)=|B|^{-1}\sum_{i\in B}D_i(x)$. Independent batch choices make the iterates execute lazy reflected random walks on increasingly fine lattices inside $[-1,1]$, so every sufficiently late block visits the whole interval. The proposed ML guard replaces $\alpha_k$ by an effective scale $\eta_k$ satisfying $\sum_k\eta_k=\infty$ but $\sum_k\eta_k^2<\infty$, for example $\eta_k=\eta_0/(k+1)^p$ with $1/2<p\le1$, or applies this schedule only after a flat-region detector fires. The square-summability condition does not by itself guarantee neural-network convergence, but directly removes the infinite accumulated-variance regime exploited by the counterexample.

Implementation notes

1. Integration point: modify the optimizer's scalar learning-rate multiplier immediately before the parameter update, not the gradient computation. For AdamW, SGD, or a subgradient optimizer, let $u_k$ be the proposed preconditioned update and apply $\Delta\theta_k=-\eta_k u_k$. Maintain a separate global step $k$, and optionally maintain a per-layer effective-update norm. Detect a flat/noisy regime when the exponentially averaged loss change is below $\epsilon_L$, the gradient norm is below $\epsilon_g$, and the cosine similarity between consecutive minibatch gradients is near zero. 2. Pseudocode: for k: g=backward(minibatch); u=optimizer_precondition(g,state); if flat_detector: eta=eta0*(k_flat+1)^(-p); else eta=base_lr; theta <- theta - eta*u; state <- optimizer_state_update(...). Use $p=0.75$ initially, and reset $k_flat$ only after a sustained non-flat interval. Apply weight decay separately so the test isolates stochastic-update accumulation. Log the trajectory norm, late-window parameter variance, loss variance, and minibatch-gradient disagreement. 3. Computed from the paper: the failure condition is the coexistence of vanishing stepsizes with divergent $\sum\alpha_k^2$, and the flat-set example demonstrates that objective convergence does not imply iterate convergence. Estimated empirically: the flat detector, gradient covariance, and whether late checkpoints span a broad low-loss region. The square-summable schedule is known analytically from the p-series test; no Hessian or exact critical set is required. 4. First experiment: train a small two-layer ReLU MLP on MNIST or CIFAR-10 with SGD and minibatches of 32, comparing cosine decay, inverse-square-root decay $p=1/2$, and the proposed guarded schedule with $p=0.75$. Run long enough for the training loss to plateau, use five seeds, and compare final loss/accuracy with the variance of parameters and predictions across the last 20 checkpoints. The expected signal is substantially lower late-training parameter and prediction wandering, with no material loss in accuracy; a stronger result is convergence of checkpoint predictions where the baseline continues to move at nearly constant training loss.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.