# Mean-Square-Stable Noise Homotopy

- ID: 2479
- Canonical URL: https://synthcore.org/idea/2479/mean-square-stable-noise-homotopy
- API JSON: https://synthcore.org/api/idea/2479.json
- API Markdown: https://synthcore.org/api/idea/2479.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.17940](https://arxiv.org/abs/2608.17940)
- Category: dynamics
- Solves: stability, sample-efficiency
- ML areas: optimizer, scheduler, training-dynamics, regularization
- Math tags: control-theory, dynamical-systems, stochastic-processes, spectral-theory
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Train with a continuation parameter that gradually increases stochasticity, such as dropout, augmentation magnitude, gradient noise, or temperature, while monitoring the local mean-square stability of the parameter update. The network first solves a low-noise problem with a larger stability margin and is then continued toward the desired noisy objective instead of entering a high-noise regime abruptly.

## Mathematical statement

The paper considers dynamics of the form dx = (A x + B u) dt + (C x + D u) dW and preserves mean-square stability of policy iterates. For neural optimization, let z_k be a perturbation around the current parameter trajectory, let F_s be the Jacobian of the deterministic update at diffusion scale s, and let G_{s,r} be the Jacobian associated with stochastic channel r. The linearized stochastic update is z_{k+1} = (F_s + sum_r G_{s,r} xi_{k,r}) z_k, where xi_{k,r} are independent zero-mean unit-variance noises. Its second-moment dynamics are governed by M_s = F_s tensor F_s + sum_r G_{s,r} tensor G_{s,r}. Mean-square contraction is expected when the spectral radius rho(M_s) is less than one. The continuation scheduler increases s only while the estimated stability margin 1 - rho(M_s) remains above a chosen safety margin delta.

## Key formulas

- $$dx(t)=\left(Ax(t)+\sum_i B_i u_i(t)\right)dt+\left(Cx(t)+\sum_i D_i u_i(t)\right)dW(t)$$
- $$z_{k+1}=\left(F_s+\sum_{r=1}^{R}G_{s,r}\xi_{k,r}\right)z_k$$
- $$\mathcal{M}_s=F_s\otimes F_s+\sum_{r=1}^{R}G_{s,r}\otimes G_{s,r},\qquad \rho(\mathcal{M}_s)<1$$
- $$s_{k+1}=\min(s_{\max},s_k+\Delta s)\quad\text{if}\quad \widehat{\rho}(\mathcal{M}_{s_{k+1}})\leq 1-\delta$$

## Implementation notes

Integration point: wrap SGD, Adam, or AdamW with a scalar continuation variable s. Define the target stochasticity, such as dropout probability p, augmentation magnitude a, or explicit gradient-noise variance, as s times its final value. Start at s = 0 or a small positive value and increase it only after the local update dynamics have a measured contraction margin. Pseudocode: initialize s and train for H steps; save the current parameters; generate several stochastic updates from identical parameters; estimate the deterministic update Jacobian F_s with finite-difference Jacobian-vector products or automatic-differentiation JVPs; estimate stochastic channel operators G_{s,r} from update differences across dropout masks or minibatches; use power iteration on the Kronecker operator M_s, or use the cheaper empirical second-moment ratio m_hat = mean(||z_next|| squared) divided by mean(||z|| squared); if m_hat is below 1 minus delta, increase s by Delta s; otherwise hold s and reduce the learning rate. The paper provides the mean-square-stability and homotopy mechanism; all neural Jacobians are estimated empirically. First cheap experiment: train a three-layer MLP on MNIST with SGD and dropout p = 0.3. Compare abrupt dropout activation against a 20-epoch homotopy from p = 0 to p = 0.3 across learning rates. The prediction is that squared perturbation growth changes from decay to growth near estimated rho(M_s) = 1, and the measured divergence boundary should agree with this estimate within approximately 20 percent.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a reproducible MVP in experiment.py with scalar mean-square stability verification, parameter sweeps, and an abrupt-dropout versus scheduled-dropout digits MLP experiment. The mechanism manifested in the toy system: the Monte Carlo second-moment ratio matched the predicted spectral factor within 0.28% maximum relative error, and the swept stability boundaries matched the analytic predictions. The neural experiment did not show a practical win: final homotopy accuracy was 94.67% versus 95.11% for abrupt dropout, so the result supports the stability mechanism but not improved task performance in this tiny run.

### Mechanism check

- Verdict: Built a reproducible MVP in experiment.py with scalar mean-square stability verification, parameter sweeps, and an abrupt-dropout versus scheduled-dropout digits MLP experiment. The mechanism manifested in the toy system: the Monte Carlo second-moment ratio matched the predicted spectral factor within 0.28% maximum relative error, and the swept stability boundaries matched the analytic predictions. The neural experiment did not show a practical win: final homotopy accuracy was 94.67% versus 95.11% for abrupt dropout, so the result supports the stability mechanism but not improved task performance in this tiny run.
- Confidence: 7/10
- Limitations: The neural comparison uses one fixed seed, one small digits dataset, one learning rate, and a fixed linear continuation rather than a full adaptive neural Jacobian scheduler. The toy boundary observations use dense grids evaluated from the same scalar formula, validating the formula and scaling rather than independently estimating a neural-network stability boundary. No MNIST or multi-seed robustness study was run.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.241
- Paired wins: 2/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/848/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/848/bench_report.json)
- [experiment.py](https://synthcore.org/code/848/experiment.py)
- [report.md](https://synthcore.org/code/848/report.md)
- [report_bench_2026-09-01T122748.md](https://synthcore.org/code/848/report_bench_2026-09-01T122748.md)
- [results.json](https://synthcore.org/code/848/results.json)
- [Download all files as ZIP](https://synthcore.org/download/848)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.