# Cubic-Rate Third-Order Langevin Optimizer

- ID: 1204
- Canonical URL: https://synthcore.org/idea/1204/cubic-rate-third-order-langevin-optimizer
- API JSON: https://synthcore.org/api/idea/1204.json
- API Markdown: https://synthcore.org/api/idea/1204.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.20882](https://arxiv.org/abs/2607.20882)
- Category: dynamics
- Solves: speedup, stability, sample-efficiency
- ML areas: optimizer, training-dynamics, sampling
- Math tags: stochastic-processes, dynamical-systems, spectral-theory, probability, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace the usual parameter-plus-momentum Langevin state with a three-level chain consisting of parameters, velocity, and acceleration, while injecting Gaussian noise only into the highest auxiliary state. At a saddle, the escaping direction has a positive rate given by a cubic characteristic equation; use this rate to choose damping or adapt the temperature so that basin escape is accelerated without making the dynamics unstable.

## Mathematical statement

The paper defines the Gibbs target $\pi_{\varepsilon}^{\theta}(d\theta)=Z_{\varepsilon}^{-1}e^{-U(\theta)/\varepsilon}d\theta$, where $U:\mathbb{R}^{d}\to\mathbb{R}$ is the objective, $\varepsilon>0$ is temperature, and $Z_{\varepsilon}$ is the normalizing constant. It contrasts this with overdamped Langevin, $d\theta_t=-\nabla U(\theta_t)dt+\sqrt{2\varepsilon}dB_t$, where $B_t$ is standard Brownian motion. The transferable adaptation is the third-order chain $dx_t=v_tdt$, $dv_t=a_tdt$, $da_t=-\nabla U(x_t)dt-\gamma a_tdt+\sqrt{2\gamma\varepsilon}\,dB_t$, with parameter $x_t$, velocity $v_t$, acceleration $a_t$, damping $\gamma>0$, and Brownian motion $B_t$. Linearizing along a Hessian eigenvector with eigenvalue $h$ gives the characteristic polynomial $p_h(\lambda)=\lambda^3+\gamma\lambda^2+h$. At an index-one saddle, $h=-\kappa<0$, so the unique positive unstable rate $r(\gamma,\kappa)$ satisfies $r^3+\gamma r^2-\kappa=0$. The paper also gives the weak capacity-hitting identity $\int_{\partial A_\varepsilon}\mathbb{E}_z[\tau_{S_\varepsilon}]\nu_\varepsilon(dz)=\operatorname{Cap}_\varepsilon(A_\varepsilon,S_\varepsilon)^{-1}\int h_\varepsilon^*(z)\pi_\varepsilon(dz)$, where $A_\varepsilon$ and $S_\varepsilon$ are source and target wells, $\tau_{S_\varepsilon}$ is a hitting time, $h_\varepsilon^*$ is the adjoint committor, $\nu_\varepsilon$ is the weak equilibrium measure, and $\operatorname{Cap}_\varepsilon$ is weak capacity. Its intrawell asymptotic is $\int h_\varepsilon^*d\pi_\varepsilon\sim Z_\varepsilon^{-1}(2\pi\varepsilon)^{3d/2}[L^d\sqrt{\det\nabla^2U(m)}]^{-1}e^{-U(m)/\varepsilon}$, where $m$ is a local minimum and $L$ is a kinetic normalization constant. The implementation uses the cubic unstable rate as a local curvature-aware integration and escape diagnostic; the full capacity prefactor is not assumed to transfer exactly to minibatch training.

## Key formulas

- $$\pi_{\varepsilon}^{\theta}(d\theta)=Z_{\varepsilon}^{-1}e^{-U(\theta)/\varepsilon}\,d\theta$$
- $$d\theta_t=-\nabla U(\theta_t)dt+\sqrt{2\varepsilon}\,dB_t$$
- $$dx_t=v_tdt,\qquad dv_t=a_tdt,\qquad da_t=-\nabla U(x_t)dt-\gamma a_tdt+\sqrt{2\gamma\varepsilon}\,dB_t$$
- $$p_h(\lambda)=\lambda^3+\gamma\lambda^2+h,\qquad r^3+\gamma r^2-\kappa=0\quad(h=-\kappa<0),\qquad r>0$$

## Implementation notes

Integrate the method into the optimizer state update, replacing SGD with momentum or Adam's single velocity tensor by two fp32 tensors $v$ and $a$ matching the parameter shapes. For each minibatch, compute $g=\nabla_x\ell(x)$ and use it as an estimate of $\nabla U(x)$. With step size $dt$, damping $\gamma$, and temperature $\varepsilon$, use the following update: sample $\xi\sim\mathcal{N}(0,I)$; set $a\leftarrow a+dt(-g-\gamma a)+\sqrt{2\gamma\varepsilon dt}\,\xi$; set $v\leftarrow v+dt\,a$; set $x\leftarrow x+dt\,v$. Noise must enter only the acceleration state. Weight decay can be applied separately to $g$, and gradient clipping should be tested only as a controlled stability baseline. Every $K$ optimizer steps, estimate the most negative local curvature using 3--5 Hessian-vector products and a small Lanczos iteration. If the smallest estimated Hessian eigenvalue is $\lambda_{\min}$, define $\kappa=\max(0,-\lambda_{\min})$. Solve $r^3+\gamma r^2-\kappa=0$ by bisection on $[0,\kappa^{1/3}+\sqrt{\kappa/\gamma}+1]$, or use Newton iteration initialized at $\max(\kappa^{1/3},10^{-3})$. Enforce $dt\,r\leq0.2$ for explicit integration; reduce $dt$ when this bound is violated. A more ambitious controller changes $\gamma$ to target $dt\,r\approx0.1$. The SDE and cubic relation come from the paper; the curvature estimate and damping controller are empirical adaptations. Begin with a small MLP on a synthetic four-well objective or two-dimensional moons classification, then test a small ResNet on CIFAR-10. Match all methods by gradient evaluations and wall-clock time against SGD, SGD with momentum, and underdamped Langevin. Record loss descent, escape time from deliberately poor basins, parameter-norm growth, gradient variance, and test accuracy across at least 20 seeds. Success is faster escape or lower loss at equal compute without exploding $\|a\|$; failure is loss divergence, NaNs, or no improvement over momentum after tuning.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a PyTorch third-order Langevin optimizer with velocity and acceleration states, acceleration-only Gaussian noise, cubic-rate solver, and optional curvature-based step-size limiting. The mechanism checks succeeded: measured unstable rates matched the cubic root within 0.04%, curvature-scaling slopes matched (0.33730 observed vs 0.33720 predicted), rates increased monotonically with negative curvature, and dt·r≈0.2 produced only 5.66% rate error. The optimizer did not win the toy MLP benchmark: momentum reached loss 0.0516 versus 1.7160 for third-order Langevin after 300 steps.

### Mechanism check

- Verdict: Built a PyTorch third-order Langevin optimizer with velocity and acceleration states, acceleration-only Gaussian noise, cubic-rate solver, and optional curvature-based step-size limiting. The mechanism checks succeeded: measured unstable rates matched the cubic root within 0.04%, curvature-scaling slopes matched (0.33730 observed vs 0.33720 predicted), rates increased monotonically with negative curvature, and dt·r≈0.2 produced only 5.66% rate error. The optimizer did not win the toy MLP benchmark: momentum reached loss 0.0516 versus 1.7160 for third-order Langevin after 300 steps.
- Confidence: 9/10
- Limitations: Only a scalar linearized saddle and one small synthetic full-batch MLP were tested; no 20-seed study, minibatch training, CIFAR-10, Hessian-vector-product curvature estimator, adaptive damping controller, sampling-distribution validation, or systematic hyperparameter tuning was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/487/bench_report.json)
- [benchmark_mlp.py](https://synthcore.org/code/487/benchmark_mlp.py)
- [benchmark_results.json](https://synthcore.org/code/487/benchmark_results.json)
- [report.md](https://synthcore.org/code/487/report.md)
- [report_bench_2026-08-31T153402.md](https://synthcore.org/code/487/report_bench_2026-08-31T153402.md)
- [results.json](https://synthcore.org/code/487/results.json)
- [stage2_local_bench.py](https://synthcore.org/code/487/stage2_local_bench.py)
- [stage2_official_bench.py](https://synthcore.org/code/487/stage2_official_bench.py)
- [third_order_optimizer.py](https://synthcore.org/code/487/third_order_optimizer.py)
- [verify_cubic_langevin.py](https://synthcore.org/code/487/verify_cubic_langevin.py)
- [Download all files as ZIP](https://synthcore.org/download/487)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.