△ Mechanism confirmed, baseline not beaten 2026

Lyapunov-Calibrated Multiplicative Noise

Implementation & benchmark of arXiv:2607.24925 — Effective Field Theory of Operator Scrambling from Strong-to-Weak Symmetry Breaking

Usefulness7/10

Difficulty4/10

Novelty6/10

Training Stability Generalization Scalability Optimizer Regularization Training dynamics Transformer Rnn Dynamical systems Control theory Stochastic processes Linear algebra

Source paper: Effective Field Theory of Operator Scrambling from Strong-to-Weak Symmetry Breaking arXiv:2607.24925 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use measured local Jacobian growth to set the variance of dropout, feature noise, or stochastic-depth perturbations, implementing the paper's fluctuation-response idea that multiplicative noise is tied to the positive scrambling or Lyapunov rate. The controller maintains a target growth regime instead of applying a fixed noise schedule throughout training. It predicts a stability transition when the estimated growth rate crosses zero and a variance-growth proportionality that can be tested directly.

Formulas

$$\widehat r_t=\frac{1}{L}\sum_{\ell=1}^{L}\log\left(\frac{\|J_\ell v_\ell\|_2}{\|v_\ell\|_2}\right),\qquad v_{\ell+1}=\frac{J_\ell v_\ell}{\|J_\ell v_\ell\|_2}$$

$$\operatorname{Var}(z_i^\ell\mid u_i^\ell)=2\kappa\,[\widehat r_t-r_{\mathrm{target}}]_+\,\Delta t\,u_i^\ell\left(1-\frac{u_i^\ell}{K}\right)$$

$$u_i^{\ell+1}-u_i^\ell\approx r_{\mathrm{target}}u_i^\ell\Delta t+\sqrt{2\kappa r_{\mathrm{target}}u_i^\ell(1-u_i^\ell/K)\Delta t}\,\epsilon_i^\ell$$

$$\widehat r_t=0\text{ is the predicted amplification-to-contraction boundary}$$

Mathematical statement

Let h_{ell+1} = F_theta(h_ell) be a layer map and J_ell = partial F_theta / partial h_ell its Jacobian. Define the empirical finite-time Lyapunov rate r_hat = (1/L) sum_ell log(||J_ell v_ell||/||v_ell||), where v_ell is a normalized probe propagated by power iteration. Positive r_hat indicates amplification and negative r_hat indicates contraction. For a scalar stochastic gate z with mean one, impose conditional variance Var(z | u) = 2 kappa [r_hat-r_target]_+ Delta t u(1-u/K), where [a]_+ = max(a,0), r_target is the desired growth rate, and u is a normalized local influence variable. This transfers the paper's relation between multiplicative noise and positive Lyapunov growth into a trainable regularization controller.

Implementation notes

Add this controller to the optimizer or training loop of a residual network, Transformer, or recurrent model. Apply the stochastic gate only to residual branches, attention outputs, or hidden-state transitions, not to normalization statistics. Every M optimization steps, sample one or two random probe vectors, compute Jacobian-vector products with automatic differentiation, normalize the probes through depth, and estimate r_hat from the average logarithmic norm growth. Exponentially smooth r_hat and set q_t = clip(2 kappa [r_hat-r_target]_+ Delta t, 0, q_max). For each residual branch draw z = 1 + sqrt(q_t u(1-u/K)) epsilon, optionally renormalize z to batch mean one, and update h by h <- h + z F_theta(h). If r_hat exceeds a hard ceiling, reduce residual scale or learning rate before increasing noise. The Jacobian estimate, K, and kappa are empirical; the growth/noise proportionality and zero-growth boundary come from the paper's effective theory. The first experiment should compare fixed dropout, cosine dropout, and this controller on a 12-layer Transformer trained on WikiText-2 and a synthetic long-horizon copy task. Log validation loss, gradient norm, hidden-state norm, r_hat, and gate variance. The prediction is that instability begins near r_hat = 0 and that conditional gate variance versus [r_hat-r_target]_+u(1-u/K) has fitted slope 2 kappa Delta t within 25 percent. Long-horizon degradation should coincide with persistently positive r_hat.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 7 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a compact Lyapunov-calibrated multiplicative-noise controller and verified its mechanism numerically. Autodiff finite-time Lyapunov rates matched the exact log|1+a| prediction with maximum error 5.1e-8, including the r_hat=0 boundary at a=0; the controller slope was 0.14 versus predicted 0.14, and sampled gate variance had fitted slope 0.999 versus predicted 1.0. In the tiny regression test, adaptive noise did not improve loss over fixed noise, so the mechanism is confirmed but no training win was observed.

Agent confidence: 9/10

Baseline: Fixed q=0.08: final MSE 0.0049883, best MSE 0.0049737, mean Lyapunov rate -0.01398
Idea: Adaptive q=0.14*[r_hat]_+: final MSE 0.0049937, best MSE 0.0049937, mean Lyapunov rate -0.01412, mean q 0.000632; exact-rate max error 5.1e-8, q slope 0.1400 vs 0.1400 predicted, variance slope 0.9989 vs 1.0 predicted

Limitations: The training comparison is only a small synthetic scalar residual regression, not WikiText-2, a Transformer, or a long-horizon copy task. It does not test generalization, wall-clock overhead, hard-ceiling learning-rate intervention, smoothed noisy Jacobian estimates, or multi-layer nonlinear Jacobians.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.24925) Sep 2, 2026

No significant difference vs. the tuned baseline (+1.1%, p=0.9849, wins 5 / 7 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 7
Baseline mean: 0.0118
Idea mean: 0
Effect (Δ): 0.0001 (+1.1%; negative = idea better)
Wins: 5 / 7 paired seeds
p-value: 0.9849 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±18.2%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 9 configs

Limitations:

The final run had one non-finite adaptive seed, leaving 7 paired seeds rather than the required 8; therefore the result is not a fully valid 8-pair protocol win claim. The implementation used a local residual RNN rather than the exact bench rnn_small internals because the idea modifies the recurrent training forward pass, and no hard-ceiling learning-rate intervention or larger-scale sequence benchmark was tested.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Lyapunov-calibrated multiplicative noise in a dynamics-track residual RNN and ran matched fixed-noise versus adaptive-noise training with shared learning-rate and method-knob sweeps. The trained-model mechanism signature was confirmed: observed gate-variance slope 1.001995 versus predicted 1.0, a 0.2% relative error. The adaptive system did not improve the standard dynamics MSE: paired delta_mean was +0.00012657 with permutation p=0.98495, so there was no significant win; the comparison retained 7 finite pairs because one adaptive run was non-finite.

Baseline: Best fixed-noise configuration lr=0.006, q=0.01; full 8-seed mean MSE 0.0117805497, std 0.0035593563.
Idea: Best adaptive configuration lr=0.006, kappa=0.35; paired comparison delta_mean +0.0001265707 MSE, 5/7 paired wins, p=0.98495, no significant win. Mechanism slope 1.0019952 versus expected 1.0, confirmed=true.

Benchmark result