# Lyapunov-Calibrated Multiplicative Noise

- ID: 1380
- Canonical URL: https://synthcore.org/idea/1380/lyapunov-calibrated-multiplicative-noise
- API JSON: https://synthcore.org/api/idea/1380.json
- API Markdown: https://synthcore.org/api/idea/1380.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.24925](https://arxiv.org/abs/2607.24925)
- Category: training
- Solves: stability, generalization, scalability
- ML areas: optimizer, regularization, training-dynamics, transformer, rnn
- Math tags: dynamical-systems, control-theory, stochastic-processes, linear-algebra
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Use measured local Jacobian growth to set the variance of dropout, feature noise, or stochastic-depth perturbations, implementing the paper's fluctuation-response idea that multiplicative noise is tied to the positive scrambling or Lyapunov rate. The controller maintains a target growth regime instead of applying a fixed noise schedule throughout training. It predicts a stability transition when the estimated growth rate crosses zero and a variance-growth proportionality that can be tested directly.

## Mathematical statement

Let h_{ell+1} = F_theta(h_ell) be a layer map and J_ell = partial F_theta / partial h_ell its Jacobian. Define the empirical finite-time Lyapunov rate r_hat = (1/L) sum_ell log(||J_ell v_ell||/||v_ell||), where v_ell is a normalized probe propagated by power iteration. Positive r_hat indicates amplification and negative r_hat indicates contraction. For a scalar stochastic gate z with mean one, impose conditional variance Var(z | u) = 2 kappa [r_hat-r_target]_+ Delta t u(1-u/K), where [a]_+ = max(a,0), r_target is the desired growth rate, and u is a normalized local influence variable. This transfers the paper's relation between multiplicative noise and positive Lyapunov growth into a trainable regularization controller.

## Key formulas

- $$\widehat r_t=\frac{1}{L}\sum_{\ell=1}^{L}\log\left(\frac{\|J_\ell v_\ell\|_2}{\|v_\ell\|_2}\right),\qquad v_{\ell+1}=\frac{J_\ell v_\ell}{\|J_\ell v_\ell\|_2}$$
- $$\operatorname{Var}(z_i^\ell\mid u_i^\ell)=2\kappa\,[\widehat r_t-r_{\mathrm{target}}]_+\,\Delta t\,u_i^\ell\left(1-\frac{u_i^\ell}{K}\right)$$
- $$u_i^{\ell+1}-u_i^\ell\approx r_{\mathrm{target}}u_i^\ell\Delta t+\sqrt{2\kappa r_{\mathrm{target}}u_i^\ell(1-u_i^\ell/K)\Delta t}\,\epsilon_i^\ell$$
- $$\widehat r_t=0\text{ is the predicted amplification-to-contraction boundary}$$

## Implementation notes

Add this controller to the optimizer or training loop of a residual network, Transformer, or recurrent model. Apply the stochastic gate only to residual branches, attention outputs, or hidden-state transitions, not to normalization statistics. Every M optimization steps, sample one or two random probe vectors, compute Jacobian-vector products with automatic differentiation, normalize the probes through depth, and estimate r_hat from the average logarithmic norm growth. Exponentially smooth r_hat and set q_t = clip(2 kappa [r_hat-r_target]_+ Delta t, 0, q_max). For each residual branch draw z = 1 + sqrt(q_t u(1-u/K)) epsilon, optionally renormalize z to batch mean one, and update h by h <- h + z F_theta(h). If r_hat exceeds a hard ceiling, reduce residual scale or learning rate before increasing noise. The Jacobian estimate, K, and kappa are empirical; the growth/noise proportionality and zero-growth boundary come from the paper's effective theory. The first experiment should compare fixed dropout, cosine dropout, and this controller on a 12-layer Transformer trained on WikiText-2 and a synthetic long-horizon copy task. Log validation loss, gradient norm, hidden-state norm, r_hat, and gate variance. The prediction is that instability begins near r_hat = 0 and that conditional gate variance versus [r_hat-r_target]_+u(1-u/K) has fitted slope 2 kappa Delta t within 25 percent. Long-horizon degradation should coincide with persistently positive r_hat.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a compact Lyapunov-calibrated multiplicative-noise controller and verified its mechanism numerically. Autodiff finite-time Lyapunov rates matched the exact log|1+a| prediction with maximum error 5.1e-8, including the r_hat=0 boundary at a=0; the controller slope was 0.14 versus predicted 0.14, and sampled gate variance had fitted slope 0.999 versus predicted 1.0. In the tiny regression test, adaptive noise did not improve loss over fixed noise, so the mechanism is confirmed but no training win was observed.

### Mechanism check

- Verdict: Built a compact Lyapunov-calibrated multiplicative-noise controller and verified its mechanism numerically. Autodiff finite-time Lyapunov rates matched the exact log|1+a| prediction with maximum error 5.1e-8, including the r_hat=0 boundary at a=0; the controller slope was 0.14 versus predicted 0.14, and sampled gate variance had fitted slope 0.999 versus predicted 1.0. In the tiny regression test, adaptive noise did not improve loss over fixed noise, so the mechanism is confirmed but no training win was observed.
- Confidence: 9/10
- Limitations: The training comparison is only a small synthetic scalar residual regression, not WikiText-2, a Transformer, or a long-horizon copy task. It does not test generalization, wall-clock overhead, hard-ceiling learning-rate intervention, smoothed noisy Jacobian estimates, or multi-layer nonlinear Jacobians.

### Practical benchmark

- Paired seeds: 7
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.98495
- Paired wins: 5/7
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/537/bench_report.json)
- [bench_run.py](https://synthcore.org/code/537/bench_run.py)
- [report.md](https://synthcore.org/code/537/report.md)
- [report_bench_2026-08-31T172101.md](https://synthcore.org/code/537/report_bench_2026-08-31T172101.md)
- [results.json](https://synthcore.org/code/537/results.json)
- [run_experiment.py](https://synthcore.org/code/537/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/537)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.