# Square-Summable Noise Guard

- ID: 2557
- Canonical URL: https://synthcore.org/idea/2557/square-summable-noise-guard
- API JSON: https://synthcore.org/api/idea/2557.json
- API Markdown: https://synthcore.org/api/idea/2557.md
- Verification status: unverified
- Source: [arXiv:2608.19074](https://arxiv.org/abs/2608.19074)
- Category: training
- Solves: stability, generalization
- ML areas: optimizer, training-dynamics, loss
- Math tags: stochastic-processes, probability, convex-analysis, dynamical-systems
- Ratings: usefulness 5/10; difficulty 3/10; novelty 3/10

## Idea description

Add a late-training safeguard that decays the effective stochastic update scale fast enough to make the accumulated update variance finite. The safeguard is motivated by the paper's bounded reflected-random-walk counterexample: iterates can keep traversing an entire flat critical set forever even though the stepsize tends to zero and the objective values remain optimal.

## Mathematical statement

The paper studies the finite-sum objective \(J(w)=\frac{1}{n}\sum_{i=1}^n f_i(w)\) and the mini-batch stochastic approximation update \(w_{k+1}=w_k-\alpha_k d_k\), where \(B_k\) is an independently sampled nonempty mini-batch and \(d_k\in |B_k|^{-1}\sum_{i\in B_k}D_i(w_k)\). Its counterexample uses \(\alpha_k=o(1/\log k)\), yet the iterates do not converge. In the explicit two-summand construction, the averaged field is \(D_J(x)=\partial_C\phi(x)\), with \(D_J(x)=\{0\}\) for \(-1<x<1\), and the critical set is \(\operatorname{crit}_J=[-1,1]\) with \(J|_{[-1,1]}\equiv0\). For each batch \(B\in\mathcal B=\{\{1\},\{2\},\{1,2\}\}\), the selected direction is the minimum-norm element \(g_B(x)=\operatorname{proj}_{G_B(x)}(0)\), where \(G_B(x)=|B|^{-1}\sum_{i\in B}D_i(x)\). Independent batch choices make the iterates execute lazy reflected random walks on increasingly fine lattices inside \([-1,1]\), so every sufficiently late block visits the whole interval. The proposed ML guard replaces \(\alpha_k\) by an effective scale \(\eta_k\) satisfying \(\sum_k\eta_k=\infty\) but \(\sum_k\eta_k^2<\infty\), for example \(\eta_k=\eta_0/(k+1)^p\) with \(1/2<p\le1\), or applies this schedule only after a flat-region detector fires. The square-summability condition does not by itself guarantee neural-network convergence, but directly removes the infinite accumulated-variance regime exploited by the counterexample.

## Key formulas

- $$J(w)=\frac{1}{n}\sum_{i=1}^{n}f_i(w),\qquad w_{k+1}=w_k-\alpha_k d_k,\qquad d_k\in\frac{1}{|B_k|}\sum_{i\in B_k}D_i(w_k).$$
- $$\alpha_k=o(1/\log k),\qquad \sum_{k=0}^{\infty}\alpha_k^2=\infty.$$
- $$D_J(x)=\partial_C\phi(x)=\begin{cases}\{-1\},&x<-1,\\[-1,0],&x=-1,\\\{0\},&-1<x<1,\\[0,1],&x=1,\\\{1\},&x>1,\end{cases}\qquad \operatorname{crit}_J=[-1,1],\quad J|_{[-1,1]}\equiv0.$$
- $$\eta_k=\eta_0(k+1)^{-p},\qquad \frac12<p\le1,\qquad \sum_k\eta_k=\infty,\quad \sum_k\eta_k^2<\infty,\qquad w_{k+1}=w_k-\eta_k d_k.$$

## Implementation notes

1. Integration point: modify the optimizer's scalar learning-rate multiplier immediately before the parameter update, not the gradient computation. For AdamW, SGD, or a subgradient optimizer, let \(u_k\) be the proposed preconditioned update and apply \(\Delta\theta_k=-\eta_k u_k\). Maintain a separate global step \(k\), and optionally maintain a per-layer effective-update norm. Detect a flat/noisy regime when the exponentially averaged loss change is below \(\epsilon_L\), the gradient norm is below \(\epsilon_g\), and the cosine similarity between consecutive minibatch gradients is near zero. 2. Pseudocode: `for k: g=backward(minibatch); u=optimizer_precondition(g,state); if flat_detector: eta=eta0*(k_flat+1)^(-p); else eta=base_lr; theta <- theta - eta*u; state <- optimizer_state_update(...)`. Use \(p=0.75\) initially, and reset \(k_flat\) only after a sustained non-flat interval. Apply weight decay separately so the test isolates stochastic-update accumulation. Log the trajectory norm, late-window parameter variance, loss variance, and minibatch-gradient disagreement. 3. Computed from the paper: the failure condition is the coexistence of vanishing stepsizes with divergent \(\sum\alpha_k^2\), and the flat-set example demonstrates that objective convergence does not imply iterate convergence. Estimated empirically: the flat detector, gradient covariance, and whether late checkpoints span a broad low-loss region. The square-summable schedule is known analytically from the p-series test; no Hessian or exact critical set is required. 4. First experiment: train a small two-layer ReLU MLP on MNIST or CIFAR-10 with SGD and minibatches of 32, comparing cosine decay, inverse-square-root decay \(p=1/2\), and the proposed guarded schedule with \(p=0.75\). Run long enough for the training loss to plateau, use five seeds, and compare final loss/accuracy with the variance of parameters and predictions across the last 20 checkpoints. The expected signal is substantially lower late-training parameter and prediction wandering, with no material loss in accuracy; a stronger result is convergence of checkpoint predictions where the baseline continues to move at nearly constant training loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.