# Ward-Calibrated Training Noise

- ID: 2745
- Canonical URL: https://synthcore.org/idea/2745/ward-calibrated-training-noise
- API JSON: https://synthcore.org/api/idea/2745.json
- API Markdown: https://synthcore.org/api/idea/2745.md
- Verification status: unverified
- Source: [arXiv:2608.26468](https://arxiv.org/abs/2608.26468)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: optimizer, training-dynamics, scheduler, regularization
- Math tags: stochastic-processes, statistical-mechanics, dynamical-systems, control-theory, probability
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Treat a slowly varying block of neural-network parameters as a coarse-grained stochastic process and continuously estimate both its covariance spectrum and its linear response to small artificial perturbations. Use the fluctuation–response mismatch as a feedback signal to tune injected parameter noise or minibatch size; the thermal Einstein relation is imposed only when a calibrated equilibrium-like regime is desired, while antisymmetric response components are retained as admissible nonequilibrium probability currents.

## Mathematical statement

Let \(\theta_t\in\mathbb{R}^d\) denote parameters or a low-dimensional projection of parameters, modeled by \(d\theta=b(\theta)dt+B\,dW_t\), where \(b\) is drift, \(W_t\) is Brownian motion, and \(D=BB^{\mathsf T}/2\) is the diffusion tensor. Apply a small impulse \(h(t)u\) to the update and estimate the retarded response \(R(t)=\delta\langle\theta(t)\rangle/\delta h(0)\), with Fourier transform \(R(\omega)\). Let \(C(\omega)=\langle\delta\theta(\omega)\delta\theta(-\omega)^{\mathsf T}\rangle\) be the symmetric fluctuation spectrum. In the high-temperature Einstein/FDT limit, the Ward identity becomes \(C(\omega)=2T\operatorname{Im}R(\omega)/\omega\), where \(T\) is the effective noise temperature. Away from this limit, the mismatch \(E(\omega)=C(\omega)-2T\operatorname{Im}R(\omega)/\omega\) measures nonequilibrium behavior and should not automatically be penalized. For a linearized optimizer \(d\theta=-A\theta dt+B dW\), stationarity requires \(\operatorname{Re}\lambda_k(A)>0\), and the covariance satisfies the Lyapunov equation \(A\Sigma+\Sigma A^{\mathsf T}=BB^{\mathsf T}\).

## Key formulas

- $$d\theta_t=b(\theta_t)\,dt+B\,dW_t,\qquad D=\frac{1}{2}BB^{\mathsf T}.$$
- $$R_{ij}(t)=\left.\frac{\delta\,\mathbb{E}[\theta_i(t)]}{\delta h_j(0)}\right|_{h=0},\qquad C_{ij}(\omega)=\left\langle\delta\theta_i(\omega)\delta\theta_j(-\omega)\right\rangle.$$
- $$C(\omega)=\frac{2T}{\omega}\operatorname{Im}R(\omega),\qquad E(\omega)=C(\omega)-\frac{2T}{\omega}\operatorname{Im}R(\omega).$$
- $$d\theta=-A\theta\,dt+B\,dW_t\quad\Longrightarrow\quad A\Sigma+\Sigma A^{\mathsf T}=BB^{\mathsf T},\qquad \operatorname{Re}\lambda_k(A)>0.$$

## Implementation notes

Integrate the mechanism at the optimizer level, initially on a small MLP or one transformer block rather than the full parameter vector. Maintain an orthonormal random projection matrix \(U\in\mathbb{R}^{p\times r}\), with \(r=8\) to \(32\), and record projected parameter coordinates \(z_t=U^{\mathsf T}(\theta_t-\bar\theta)\) every optimizer step. Every \(K=100\) to \(500\) steps, create a shadow copy of the optimizer state and apply a one-step impulse \(h u\) to the projected update, using a fresh random direction \(u\). Run perturbed and control trajectories for a short horizon and estimate \(\widehat R(\tau)=[z^{\mathrm{pert}}_{t+\tau}-z^{\mathrm{control}}_{t+\tau}]/h\). Estimate \(\widehat C(\omega)\) from a windowed FFT of the recorded \(z_t\). Fit \(\widehat T\) by least squares over low frequencies, then compute the scalar mismatch \(m=\sum_\omega\|\widehat C(\omega)-2\widehat T\operatorname{Im}\widehat R(\omega)/\omega\|_F^2\). Adapt the injected noise scale \(q\) using \(\log q\leftarrow\log q-\alpha(m-m_0)\), clipped to preset bounds; alternatively adapt minibatch size to control gradient-noise variance. The paper supplies the Ward/FDT structure, but response, covariance, stationarity, and temperature are estimated empirically. First experiment: train an MLP on MNIST with SGD, SGD plus fixed Langevin noise, and the adaptive controller. Use \(r=16\), \(K=200\), and several impulse sizes. The predicted signature is a measurable reduction of low-frequency mismatch \(E(\omega)\) by at least 30% in the calibrated regime. In a linear quadratic toy model, the stability boundary must remain \(\operatorname{Re}\lambda_k(A)=0\): noise adaptation can change covariance but cannot stabilize an unstable drift.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.