△ Mechanism confirmed, baseline not beaten 2026

Mean-Reverting Levy-Jump Optimizer

Implementation & benchmark of arXiv:2607.20997 — Solow system driven by $α$-stable Lévy process

Usefulness7/10

Difficulty5/10

Novelty7/10

Dynamics Stability Accuracy Generalization Optimizer Training dynamics Regularization Dynamical systems Stochastic processes Probability Control theory

Source paper: Solow system driven by $α$-stable Lévy process arXiv:2607.20997 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace purely Gaussian optimizer noise with symmetric alpha-stable jumps and add a restoring drift toward an exponential-moving-average parameter anchor. The drift prevents persistent parameter diffusion, while heavy-tailed jumps provide rare, large excursions that can cross sharp basin barriers and remain effective when gradient-noise variance is undefined.

Formulas

$$\mathbb{E}\left[e^{iu(L_\alpha(t+h)-L_\alpha(t))}\right]=\exp\left(-h\,\sigma^\alpha |u|^\alpha\right),\qquad 1<\alpha<2.$$

$$dX_t=-\lambda X_t\,dt+\sigma\,dL_\alpha(t),\qquad \phi_X(u)=\exp\left[-\frac{\sigma^\alpha}{\alpha\lambda}|u|^\alpha\right].$$

$$\theta_{t+1}=\theta_t-\eta g_t-\eta\lambda(\theta_t-m_t)+\sigma_\eta z_t,\qquad m_{t+1}=\beta m_t+(1-\beta)\theta_{t+1}.$$

$$|1-\eta\lambda|<1\quad\Longleftrightarrow\quad 0<\eta\lambda<2.$$

Mathematical statement

Let L_alpha(t) be a symmetric alpha-stable Levy process with tail index alpha in (1,2), scale sigma, and increment characteristic function E[exp(iu(L_alpha(t+h)-L_alpha(t)))] = exp(-h sigma^alpha |u|^alpha). For a scalar deviation X_t from an anchor, the model is dX_t = -lambda X_t dt + sigma dL_alpha(t), where lambda > 0 is the restoring rate. Its stationary characteristic function is phi_X(u) = exp(-sigma^alpha |u|^alpha/(alpha lambda)), so X has an alpha-stable stationary distribution and no finite second moment. In a neural optimizer, theta is the parameter vector, m is an EMA anchor, g_t is the minibatch gradient, eta is the learning rate, lambda is the anchor-restoring coefficient, and z_t is an isotropic symmetric alpha-stable vector. Use theta_{t+1} = theta_t - eta g_t - eta lambda(theta_t-m_t) + sigma_eta z_t and m_{t+1} = beta m_t + (1-beta)theta_{t+1}. Ignoring gradients and anchor motion, deterministic deviations contract when |1-eta lambda| < 1, equivalently 0 < eta lambda < 2; stable noise changes the stationary scale but not this linear contraction boundary.

Implementation notes

1. Integration point: implement this as an SGD or Adam wrapper after minibatch gradients are computed and before parameters are updated. Maintain a detached EMA tensor m for every parameter tensor. Initially use coordinate-wise symmetric alpha-stable noise. Generate each noise coordinate with a Chambers–Mallows–Stuck sampler, using a uniform angle V in (-pi/2, pi/2) and an independent exponential random variable W. Use a common alpha initially and set the jump scale sigma_eta per parameter group. 2. Pseudocode: compute g = grad(loss, theta); update m = beta*m + (1-beta)*theta; sample z = StableSymmetric(alpha, scale=1); then update theta = theta - eta*g - eta*lambda*(theta-m) + sigma_eta*z. Clip only the deterministic gradient in the first experiment, not the stable jump. 3. Computed from the paper: the stable increment law, mean-reversion structure, stationary characteristic function, and contraction condition. Estimated empirically: alpha, sigma_eta, lambda, and effective stationary scale. Estimate alpha from the empirical characteristic function or log-log tail probabilities rather than sample variance, which is infinite for alpha < 2. 4. First cheap experiment: train a small MLP on MNIST and a two-layer CNN on CIFAR-10 using SGD, Gaussian-noise SGD, and this optimizer. Sweep q = eta*lambda over 0.25, 0.5, 1, 1.5, 1.9, 2.1 in a gradient-free toy system and during training. The predicted signature is bounded median absolute deviation for q < 2 and rapid growth or oscillatory divergence for q > 2; the measured boundary should be within 20% of q_c = 2. Fit the stationary characteristic function and test whether log|phi(u)| is linear in -|u|^alpha. Compare escape from deliberately initialized sharp minima and robustness to artificial gradient spikes using median loss and interquartile ranges instead of variance.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified a symmetric alpha-stable mean-reverting optimizer toy model. The predicted contraction transition occurred exactly at q=eta*lambda=2: q=1.9 contracted, q=2.0 was marginal, and q=2.1/2.5 diverged. Stationary MAD scaled linearly with jump scale (log-log slope 1.002 vs 1), the characteristic-function coefficient scaled as sigma^1.504 (predicted 1.5), and fitted characteristic-function powers were 1.198, 1.503, and 1.800 for true alpha values 1.2, 1.5, and 1.8. The Levy optimizer was worse than Gaussian noise on this smooth quadratic, so no optimization win is claimed; the mechanism itself manifested clearly.

Agent confidence: 9/10

Baseline: Gaussian quadratic optimizer: median final |x| = 0.0918; IQR = [0.0596, 0.1345]
Idea: Levy mean-reverting quadratic optimizer: median final |x| = 0.2607; IQR = [0.1573, 0.4236]. Mechanism checks: boundary q_c observed [1.9, 2.0, 2.1] and predicted 2; MAD slope 1.002 vs 1; CF coefficient slope 1.504 vs 1.5; CF powers 1.198/1.503/1.800 vs 1.2/1.5/1.8.

Limitations: Only a NumPy toy system and one-dimensional quadratic were tested; no MNIST/CIFAR training, PyTorch optimizer wrapper, sharp-basin escape test, gradient-spike robustness test, wall-clock comparison, or multidimensional parameter experiment was run.

How to run: python3 toy_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.20997) Sep 2, 2026

Ahead of the baseline (−1.6%) but not statistically significant (p=0.3926, wins 5 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 6.9206
Idea mean: 6.8102
Effect (Δ): -0.1103 (−1.6%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.3926 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±4.3%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched tabular Friedman regression track and mlp_tiny model were tested; vision, sequence, and dynamics tracks were not tested. The benchmark used 15 epochs and reduced data sizes (1200 train, 400 test) for runtime, tested one alpha, one lambda/beta/noise combination with three learning rates, and did not test sharp-basin escape, gradient-spike robustness, wall-clock speed, or larger neural networks.

How to run: python3 bench_levy.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the mean-reverting symmetric alpha-stable optimizer as a local PyTorch training-loop intervention on the structurally matched tabular Friedman regression track with the identical mlp_tiny architecture for baseline and idea. The idea achieved a small lower mean MSE (6.8102 vs 6.9206; delta -0.1103, 5/8 paired wins), but p=0.39255, so there was no significant benchmark win. The trained-model mechanism signature was quantitatively confirmed: predicted drift multiplier 0.99 versus observed 0.99000001, and observed characteristic-function power 1.661 versus alpha 1.5.

Baseline: Adam baseline, best lr=0.01: mean test MSE 6.920556, std 0.571610, per-seed [7.096156, 6.974558, 6.666646, 6.943192, 5.700762, 7.199984, 6.888844, 7.894307]. Baseline sweep means: lr=0.001 -> 14.000885, lr=0.003 -> 7.579636, lr=0.01 -> 6.920138.
Idea: Mean-reverting Levy optimizer, best of three parity settings (lr=0.01, lambda=1.0, beta=0.95, jump scale=0.001, alpha=1.5): mean test MSE 6.810218, std 0.341144, per-seed [6.926017, 6.900168, 6.710284, 6.759966, 6.147102, 6.569409, 7.093952, 7.374846]. Paired delta=-0.110338, 5/8 wins, permutation p=0.39255, verdict='no significant win'. Idea sweep means: lr=0.01 -> 6.824109, lr=0.001 -> 13.783733, lr=0.003 -> 7.443580.

Benchmark result