# Mpemba Spectral Restart - ID: 1512 - Canonical URL: https://synthcore.org/idea/1512/mpemba-spectral-restart - API JSON: https://synthcore.org/api/idea/1512.json - API Markdown: https://synthcore.org/api/idea/1512.md - Verification status: unverified - Source: [arXiv:2607.27998](https://arxiv.org/abs/2607.27998) - Category: dynamics - Solves: speedup, stability - ML areas: optimizer, training-dynamics, initialization, scheduler - Math tags: statistical-mechanics, stochastic-processes, dynamical-systems, spectral-theory, optimization - Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10 ## Idea description Use the slow-mode content of a neural network's local optimization dynamics to choose between a near restart and a deliberately larger restart concentrated in fast-curvature directions. The larger perturbation is predicted to recover faster when it has substantially smaller overlap with the slowest Hessian modes, producing an explicit Mpemba crossover in loss or validation recovery. ## Mathematical statement Represent the local error around a stable trained solution as $e(t)=\theta(t)-\theta_\star$ and approximate gradient-flow dynamics by $\dot e=-He$, where $H=\nabla^2 L(\theta_\star)$ is positive definite with eigenpairs $(\lambda_i,v_i)$ ordered as $0<\lambda_1\leq\cdots\leq\lambda_d$. Then $e(t)=\sum_i a_i e^{-\lambda_i t}v_i$, with modal amplitudes $a_i=v_i^Te(0)$. The slow asymptotic term is $a_1e^{-\lambda_1t}$. Two initializations A and B exhibit an optimization Mpemba effect if $L_A(0)>L_B(0)$ but $|a_{1,A}|<|a_{1,B}|$ and the loss of A crosses below that of B at a finite time. For a dominant slow mode and dominant fast mode $j$, the predicted crossing follows from equating their quadratic-mode losses. The paper's nonequilibrium network supplies the analogous mechanism through relaxation eigenmodes of a Markov generator $W$; the neural implementation replaces $W$ by the local Hessian or, for activations, the Jacobian of the update map. ## Key formulas - $$\dot p(t)=Wp(t),\qquad p(t)=\pi+\sum_{k\geq 1}c_k e^{-\alpha_k t}r_k,$$ - $$\dot e=-He,\qquad e(t)=\sum_{i=1}^{d}a_i e^{-\lambda_i t}v_i,\qquad a_i=v_i^{\mathsf T}e(0),$$ - $$L(t)-L_\star\simeq\frac{1}{2}\sum_i\lambda_i a_i^2e^{-2\lambda_i t},$$ - $$t_\times\approx\frac{1}{2(\lambda_j-\lambda_1)}\log\!\left(\frac{\lambda_j(a_{j,A}^2-a_{j,B}^2)}{\lambda_1(a_{1,B}^2-a_{1,A}^2)}\right),\qquad \lambda_j>\lambda_1.$$ ## Implementation notes Integrate this as an optimizer restart or recovery controller, not as a permanent loss term. First train a small model to a reproducible solution, save parameters $\theta_\star$, create two perturbed states, and continue training with identical full-batch gradient descent. State B is the closer perturbation with substantial slow-mode content; state A is farther in Euclidean or loss distance but is projected mostly into high-curvature directions. Practical pseudocode is: (1) estimate the smallest and several largest Hessian eigenvectors using five to twenty Lanczos Hessian-vector products; (2) draw a random vector $z$, remove its slow components using $z\leftarrow z-v_1v_1^Tz$ or $z\leftarrow z-\sum_{i\leq k}v_iv_i^Tz$; (3) set $\theta_A=\theta_\star+r_Az/\|z\|$ and $\theta_B=\theta_\star+r_Bv_1/\|v_1\|$, with $r_A>r_B$; (4) run the same optimizer from both states and log loss, gradient norm, validation score, and estimated modal amplitudes every step. The paper contributes the relaxation-mode/Mpemba mechanism; Hessian eigenvectors, curvature eigenvalues, and the local quadratic approximation are estimated empirically. Begin with a two-layer MLP on MNIST or CIFAR-10, using full-batch gradient descent and a checkpoint near a strict local minimum; compare ordinary random perturbations, slow-mode perturbations, and fast-mode perturbations. The falsifiable prediction is a recovery crossover: despite $L_A(0)>L_B(0)$, A should satisfy $L_A(t)