# Mean-Reverting Levy-Jump Optimizer

- ID: 1211
- Canonical URL: https://synthcore.org/idea/1211/mean-reverting-levy-jump-optimizer
- API JSON: https://synthcore.org/api/idea/1211.json
- API Markdown: https://synthcore.org/api/idea/1211.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.20997](https://arxiv.org/abs/2607.20997)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: optimizer, training-dynamics, regularization
- Math tags: dynamical-systems, stochastic-processes, probability, control-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace purely Gaussian optimizer noise with symmetric alpha-stable jumps and add a restoring drift toward an exponential-moving-average parameter anchor. The drift prevents persistent parameter diffusion, while heavy-tailed jumps provide rare, large excursions that can cross sharp basin barriers and remain effective when gradient-noise variance is undefined.

## Mathematical statement

Let L_alpha(t) be a symmetric alpha-stable Levy process with tail index alpha in (1,2), scale sigma, and increment characteristic function E[exp(iu(L_alpha(t+h)-L_alpha(t)))] = exp(-h sigma^alpha |u|^alpha). For a scalar deviation X_t from an anchor, the model is dX_t = -lambda X_t dt + sigma dL_alpha(t), where lambda > 0 is the restoring rate. Its stationary characteristic function is phi_X(u) = exp(-sigma^alpha |u|^alpha/(alpha lambda)), so X has an alpha-stable stationary distribution and no finite second moment. In a neural optimizer, theta is the parameter vector, m is an EMA anchor, g_t is the minibatch gradient, eta is the learning rate, lambda is the anchor-restoring coefficient, and z_t is an isotropic symmetric alpha-stable vector. Use theta_{t+1} = theta_t - eta g_t - eta lambda(theta_t-m_t) + sigma_eta z_t and m_{t+1} = beta m_t + (1-beta)theta_{t+1}. Ignoring gradients and anchor motion, deterministic deviations contract when |1-eta lambda| < 1, equivalently 0 < eta lambda < 2; stable noise changes the stationary scale but not this linear contraction boundary.

## Key formulas

- $$\mathbb{E}\left[e^{iu(L_\alpha(t+h)-L_\alpha(t))}\right]=\exp\left(-h\,\sigma^\alpha |u|^\alpha\right),\qquad 1<\alpha<2.$$
- $$dX_t=-\lambda X_t\,dt+\sigma\,dL_\alpha(t),\qquad \phi_X(u)=\exp\left[-\frac{\sigma^\alpha}{\alpha\lambda}|u|^\alpha\right].$$
- $$\theta_{t+1}=\theta_t-\eta g_t-\eta\lambda(\theta_t-m_t)+\sigma_\eta z_t,\qquad m_{t+1}=\beta m_t+(1-\beta)\theta_{t+1}.$$
- $$|1-\eta\lambda|<1\quad\Longleftrightarrow\quad 0<\eta\lambda<2.$$

## Implementation notes

1. Integration point: implement this as an SGD or Adam wrapper after minibatch gradients are computed and before parameters are updated. Maintain a detached EMA tensor m for every parameter tensor. Initially use coordinate-wise symmetric alpha-stable noise. Generate each noise coordinate with a Chambers–Mallows–Stuck sampler, using a uniform angle V in (-pi/2, pi/2) and an independent exponential random variable W. Use a common alpha initially and set the jump scale sigma_eta per parameter group. 2. Pseudocode: compute g = grad(loss, theta); update m = beta*m + (1-beta)*theta; sample z = StableSymmetric(alpha, scale=1); then update theta = theta - eta*g - eta*lambda*(theta-m) + sigma_eta*z. Clip only the deterministic gradient in the first experiment, not the stable jump. 3. Computed from the paper: the stable increment law, mean-reversion structure, stationary characteristic function, and contraction condition. Estimated empirically: alpha, sigma_eta, lambda, and effective stationary scale. Estimate alpha from the empirical characteristic function or log-log tail probabilities rather than sample variance, which is infinite for alpha < 2. 4. First cheap experiment: train a small MLP on MNIST and a two-layer CNN on CIFAR-10 using SGD, Gaussian-noise SGD, and this optimizer. Sweep q = eta*lambda over 0.25, 0.5, 1, 1.5, 1.9, 2.1 in a gradient-free toy system and during training. The predicted signature is bounded median absolute deviation for q < 2 and rapid growth or oscillatory divergence for q > 2; the measured boundary should be within 20% of q_c = 2. Fit the stationary characteristic function and test whether log|phi(u)| is linear in -|u|^alpha. Compare escape from deliberately initialized sharp minima and robustness to artificial gradient spikes using median loss and interquartile ranges instead of variance.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built and numerically verified a symmetric alpha-stable mean-reverting optimizer toy model. The predicted contraction transition occurred exactly at q=eta*lambda=2: q=1.9 contracted, q=2.0 was marginal, and q=2.1/2.5 diverged. Stationary MAD scaled linearly with jump scale (log-log slope 1.002 vs 1), the characteristic-function coefficient scaled as sigma^1.504 (predicted 1.5), and fitted characteristic-function powers were 1.198, 1.503, and 1.800 for true alpha values 1.2, 1.5, and 1.8. The Levy optimizer was worse than Gaussian noise on this smooth quadratic, so no optimization win is claimed; the mechanism itself manifested clearly.

### Mechanism check

- Verdict: Built and numerically verified a symmetric alpha-stable mean-reverting optimizer toy model. The predicted contraction transition occurred exactly at q=eta*lambda=2: q=1.9 contracted, q=2.0 was marginal, and q=2.1/2.5 diverged. Stationary MAD scaled linearly with jump scale (log-log slope 1.002 vs 1), the characteristic-function coefficient scaled as sigma^1.504 (predicted 1.5), and fitted characteristic-function powers were 1.198, 1.503, and 1.800 for true alpha values 1.2, 1.5, and 1.8. The Levy optimizer was worse than Gaussian noise on this smooth quadratic, so no optimization win is claimed; the mechanism itself manifested clearly.
- Confidence: 9/10
- Limitations: Only a NumPy toy system and one-dimensional quadratic were tested; no MNIST/CIFAR training, PyTorch optimizer wrapper, sharp-basin escape test, gradient-spike robustness test, wall-clock comparison, or multidimensional parameter experiment was run.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.39255
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_levy.py](https://synthcore.org/code/490/bench_levy.py)
- [bench_report.json](https://synthcore.org/code/490/bench_report.json)
- [report.md](https://synthcore.org/code/490/report.md)
- [report_bench_2026-08-31T154337.md](https://synthcore.org/code/490/report_bench_2026-08-31T154337.md)
- [results.json](https://synthcore.org/code/490/results.json)
- [toy_experiment.py](https://synthcore.org/code/490/toy_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/490)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.