△ Mechanism confirmed, baseline not beaten 2026

Spectral Mpemba Initialization

Implementation & benchmark of arXiv:2608.07465 — Entanglement Mpemba Effect

Usefulness7/10

Difficulty5/10

Novelty7/10

Dynamics Speedup Stability Optimizer Initialization Training dynamics Dynamical systems Spectral theory Optimization Linear algebra

Source paper: Entanglement Mpemba Effect arXiv:2608.07465 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Choose an initialization that may have worse initial loss but has a smaller projection onto the slow modes of the subsequent training dynamics. Under the same optimizer, data order, and learning rate, this initialization should overtake a lower-loss baseline after a predictable crossing time, analogous to the paper's reversal of relaxation ordering.

Formulas

$$\rho_j(t)=e^{t\mathcal L}\rho_j(0),\qquad \Delta_E(t)=E[\rho_A(t)]-E[\rho_B(t)]$$

$$\Delta_E(t)\approx\sum_k a_k e^{\lambda_k t},\qquad \operatorname{Re}\lambda_k<0$$

$$L(t)-L_\star\approx\frac12\sum_k\mu_k c_k^2e^{-2\mu_k t},\qquad c_k=v_k^\top(\theta(0)-\theta_\star)$$

$$t_\times=\frac{1}{2(\mu_f-\mu_s)}\log\left[\frac{\mu_f(c_{B,f}^2-c_{A,f}^2)}{\mu_s(c_{A,s}^2-c_{B,s}^2)}\right]$$

Mathematical statement

The paper evolves two states with the same semigroup, $\rho_j(t)=e^{t\mathcal L}\rho_j(0)$, and compares an observable through $\Delta_E(t)=E[\rho_A(t)]-E[\rho_B(t)]$. Near the stationary state, expand the difference in relaxation modes as $\Delta_E(t)\approx\sum_k a_k e^{\lambda_k t}$, where $\lambda_k$ are nonzero generator eigenvalues satisfying $\operatorname{Re}\lambda_k<0$, and $a_k$ are preparation-dependent modal coefficients. If preparation A has a worse initial observable but a smaller coefficient on the slowest mode, A can overtake B after fast modes decay. The neural analogue linearizes training near a reference solution: $\delta\theta(t)=\sum_k c_k e^{-\mu_k t}v_k$, where $\mu_k>0$ are local Hessian eigenvalues, $v_k$ are orthonormal eigenvectors, and $c_k=v_k^\top(\theta(0)-\theta_\star)$. For a locally quadratic loss, $L(t)-L_\star\approx\frac12\sum_k\mu_kc_k^2e^{-2\mu_kt}$. The smallest positive $\mu_s$ controls late-time relaxation, so reducing $\mu_sc_s^2$ can outweigh a larger initial contribution from fast modes.

Implementation notes

1. Integration point: apply the mechanism at initialization or after a short warm-start checkpoint for a small MLP or transformer block. Use exactly the same optimizer, learning rate, minibatch order, weight decay, and random data pipeline for both conditions. Construct baseline B with standard Xavier or Kaiming initialization. Construct candidate A by reducing its projection onto estimated slow training directions while allowing larger projection onto fast directions, so A may begin with higher loss. 2. Pseudocode: run 20-50 deterministic warm-up updates; estimate a low-rank Hessian using Lanczos Hessian-vector products or estimate the empirical gradient-covariance spectrum; retain eigenvectors $V_s$ associated with the smallest nonzero eigenvalues and $V_f$ associated with larger eigenvalues; sample a perturbation $\xi$; set $\delta_A=\xi-V_sV_s^\top\xi+\alpha V_fV_f^\top\xi$; normalize $\delta_A$; initialize $\theta_A=\theta_0+\delta_A$, with a matched-norm random control $\theta_B$; train both with the identical update loop. Estimate $c_{j,k}=v_k^\top(\theta_j-\theta_{\rm ref})$ and predict the crossing using the displayed two-mode formula. 3. Computed from the paper's mechanism: common dissipative dynamics, modal exponential relaxation, and dominance of the slowest mode at late times. Estimated empirically: Hessian eigenvalues, eigenvectors, modal coefficients, quadratic-loss validity, and crossing time. 4. First cheap experiment: train a width-256 two-layer MLP on a fixed 2,000-example MNIST subset using full-batch gradient descent and compare standard initialization, slow-mode-suppressed initialization, and random matched-norm controls over 50 seeds. The quantitative prediction is that candidates with smaller $\mu_sc_s^2$ but initially larger loss will cross the baseline loss curve; predicted and measured crossing times should agree within 30%, and the late-time log loss gap should approach a decay slope $-2\mu_s$. If slow-mode projection fails to predict the late-time ordering, reject the method.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a diagonal quadratic training-dynamics verifier and a small nonlinear tanh-MLP Hessian-mode experiment. The toy mechanism manifested: the crossing-time formula had 20.2% mean relative error, stronger slow-mode suppression produced earlier crossings, and measured late-time slopes matched the exact discrete prediction to 5.3e-6. In the nonlinear MLP, the candidate had 4.5x higher initial loss and 25x lower slow modal energy, but did not overtake the baseline within 250 steps; therefore the mechanism is validated in the controlled local model but not demonstrated as a practical neural initialization win here.

Agent confidence: 7/10

Baseline: Quadratic baseline initial loss 0.22; nonlinear MLP final loss 8.22e-05.
Idea: Quadratic sweep: crossing formula mean relative error 0.202; stronger slow suppression caused earlier crossing; late-slope maximum error 5.3e-06. Nonlinear MLP candidate initial loss 0.970 versus 0.0443 baseline, slow modal energy 1.64e-07 versus 4.11e-06, final loss 1.17e-04 versus 8.22e-05, and no crossing by step 250.

Limitations: The neural test used a 33-parameter full-batch tanh regression problem rather than MNIST or a larger MLP, one deterministic seed, an exact Hessian at a fitted reference, and only 250 update steps. It did not test minibatch noise, Adam, weight decay, multiple seeds, or robustness of Hessian-mode estimation away from the reference solution.

How to run: python3 spectral_mpemba.py && /home/maxwelhelp/main/bin/python3 neural_mini.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.07465) Sep 2, 2026

Significantly WORSE than the tuned baseline (+3097.3%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 2.0844
Idea mean: 66.6468
Effect (Δ): 64.5623 (+3097.3%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±98.2%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the registered structurally matched tabular track was tested; vision, sequence, and dynamics were not tested. The spectral direction estimate used a small gradient-covariance probe rather than an exact Hessian, and no separate warm-start or alternative spectral estimator was evaluated.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Built and benchmarked Spectral Mpemba Initialization on the registered tabular Friedman#1 track with the shared mlp_tiny architecture and canonical train_model path. The tuned baseline achieved mean test MSE 2.08444, while the best idea configuration achieved 66.64675; paired delta was +64.56231 with permutation p=0.0081, so the idea was significantly worse. The trained-model signature showed suppressed estimated slow projection and higher initial loss, but no practical overtaking or speedup.

Baseline: Best lr=0.01, weight_decay=0.0; mean test MSE 2.08444 over 8 paired seeds.
Idea: Best shared-grid lr=0.01, weight_decay=0.0; mean test MSE 66.64675 over 8 paired seeds; delta +64.56231; 0/8 wins; permutation p=0.0081.

Benchmark result