Lyapunov-Margin Noise Scaling

Integrate the mechanism at the optimizer step. Use SGD or Adam as the nominal update and inject Gaussian parameter noise according to the estimated Lyapunov margin. First maintain a reference checkpoint theta_star, refreshed only after a validation improvement or every fixed number of successful epochs. Compute V_t = 0.5 times the squared Euclidean distance between theta_t and theta_star, preferably normalized per layer to prevent large layers from dominating. During a warm-up interval without added noise, fit a rolling regression of delta V_t on V_t; the negative slope estimates the nominal decay rate c_hat_t. Estimate q_hat_t with matched minibatch probes: apply the same gradient update twice, once without noise and once with a unit-scale Gaussian perturbation, and divide the resulting excess increase in V by the probe scale. Then update theta using theta_{t+1} = theta_t - eta_t times g_hat_t + sigma_t times xi_t, where xi_t is standard Gaussian noise, and set sigma_t using the displayed rule with safety factor rho = 0.5. Smooth c_hat and q_hat with exponential moving averages and clip sigma. The paper provides the Lyapunov and ISS argument; the checkpoint-centered V, decay estimate, and perturbation estimate are empirical approximations. The first cheap experiment should use a two-layer MLP on MNIST, comparing no noise, fixed noise, and adaptive noise. Sweep fixed noise amplitudes and define divergence as a nonnegative long-run slope of V. Prediction: the fixed-noise stability boundary occurs near q_hat times sigma = c_hat times V, while the adaptive method should keep the fitted slope of delta V negative and close to minus (1 minus rho) times c_hat.