△ Mechanism confirmed, baseline not beaten 2026

Noncommuting Bang-Bang Optimizer

Implementation & benchmark of arXiv:2607.11877 — Slow is fast: raising barriers to accelerate thermal relaxation

Usefulness8/10

Difficulty6/10

Novelty7/10

Dynamics Speedup Stability Optimizer Training dynamics Scheduler Dynamical systems Control theory Linear algebra Optimization

Source paper: Slow is fast: raising barriers to accelerate thermal relaxation arXiv:2607.11877 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a single preconditioner with a finite set of stable update operators and switch between them during training to rotate optimization error into directions that later operators remove quickly. The controller should choose a small number of hard switches, including occasional use of a seemingly slower or less aggressive preconditioner, rather than averaging all optimizers at every step.

Formulas

$$de/dt = −Aₛ(t)e, Aᵢ = GᵢH, e(T) = exp(−Aₘτₘ)⋯exp(−A₁τ₁)e(0)$$

$$[Aᵢ,Aⱼ] = AᵢAⱼ − AⱼAᵢ, log(exp(−Aⱼτⱼ)exp(−Aᵢτᵢ)) = −(Aᵢτᵢ + Aⱼτⱼ) + (τᵢτⱼ/2)[Aⱼ,Aᵢ] + O(τ³)$$

$$[Aᵢ,Aⱼ] = 0 for all i,j implies Φ(T,0) = exp(−Σᵢ τᵢAᵢ)$$

$$eₖ₊₁ = (I − ηGₛₖĤₖ)eₖ, 0 < η < 1 / λmax(Gₛₖ^(1/2)ĤₖGₛₖ^(1/2))$$

Mathematical statement

For a local quadratic loss L(θ) = 1/2 (θ − θ*)ᵀ H (θ − θ*), define the error e = θ − θ*. Preconditioned gradient flow under mode i is de/dt = −Gᵢ H e = −Aᵢe, where H is positive definite, Gᵢ is positive definite, and Aᵢ = GᵢH has positive real eigenvalues. A piecewise-constant schedule gives e(T) = exp(−Aₘτₘ)⋯exp(−A₁τ₁)e(0). If every pair commutes, all schedules reduce to the exponential of a time-averaged generator and switching cannot outperform the corresponding static allocation under the same mode durations. If [Aᵢ,Aⱼ] ≠ 0, the Baker–Campbell–Hausdorff expansion contains a rotation term proportional to τᵢτⱼ[Aⱼ,Aᵢ]. This changes eigenvector alignment and can reduce the terminal residual even when every individual mode is stable. In discrete optimization use eₖ₊₁ = (I − ηGₛₖĤₖ)eₖ, with a conservative stability condition η multiplied by the largest eigenvalue of Gₛₖ^(1/2)ĤₖGₛₖ^(1/2) less than 1.

Implementation notes

1. Exact integration point: implement a controller around an existing optimizer using two or three update modes. Mode 1 can be diagonal Adam or RMS normalization, mode 2 a blockwise whitening preconditioner, and mode 3 a damped low-rank natural-gradient approximation. Keep network parameters unchanged; only the transformation applied to the gradient changes. Start with two modes, each damped so that its preconditioned Hessian has a controlled spectral radius.

2. Pseudocode: at checkpoint k, compute gradient gₖ and maintain recent gradients or parameter displacements. Estimate Hessian-vector products Ĥₖv using automatic differentiation or a secant approximation. For every candidate mode i, form uᵢ = Gᵢgₖ and estimate λmax of Gᵢ^(1/2)ĤₖGᵢ^(1/2) with five to ten power iterations. Reject a mode if ηλmax is greater than 0.9. For candidate bang-bang schedules such as mode 1 for a steps followed by mode 2 for b steps, predict the next K updates using the local linear model eₖ₊₁ = (I − ηGₛₖĤₖ)eₖ. Choose the schedule with the smallest predicted terminal quadratic loss, execute only its first block, and replan. Use hysteresis so that a switch occurs only when the predicted improvement exceeds a fixed threshold.

3. Computed versus estimated quantities: the commutator is computed exactly in a small matrix experiment and estimated in a neural network by randomized probes. For Gaussian probes zᵣ, estimate its squared action by the average of ||AᵢAⱼzᵣ − AⱼAᵢzᵣ||², where each matrix-vector product is implemented through Hessian-vector products and preconditioner applications. Estimate the stability ceiling by power iteration. The terminal prediction is obtained by multiplying local propagators. Compare against each static mode, a fixed convex average of the modes, ordinary Adam, and SGD at matched gradient evaluations.

4. First cheap experiment and quantitative signature: use a two-dimensional quadratic with H = R(φ)diag(1,100)R(φ)ᵀ and two diagonal preconditioners expressed in different coordinate bases. Sweep φ, learning rate, and switching duration, using exact matrix exponentials. The predicted signature is zero switching advantage when ||[G₁H,G₂H]|| is zero, and increasing terminal-error improvement as the commutator norm increases. The leading eigenvector-rotation contribution should scale as τ₁τ₂||[A₁,A₂]|| for short blocks. Then test a two-layer MLP on MNIST or CIFAR-10 against Adam, fixed mode 1, fixed mode 2, and their convexly averaged preconditioner. Record loss versus gradient count, commutator norm, switch count, and the largest stable learning rate. The mechanism is supported only if switching improves terminal loss at equal compute and the improvement collapses when the modes are replaced by commuting diagonal operators.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built an exact 2D quadratic MVP for noncommuting preconditioner flows, BCH behavior, commuting collapse, discrete stability, and a replanning hard-switch controller. The quantitative mechanism checks succeeded: the BCH coefficient approached 0.532 versus the predicted 0.5, the commuting discrepancy was 5.6e-17, and commutator magnitude correlated with BCH discrepancy at 0.989. The practical controller did not win in this setup: it won on 38.7% of random initial errors and had mean loss 1.507 versus 0.235 for the best static/average comparator.

Agent confidence: 8/10

Baseline: Best static/average mean terminal loss 0.234997; fixed two-block best static loss 0.350160 for the displayed initial error.
Idea: Hard-switch controller mean terminal loss 1.507317, win rate 0.386667, mean switches 0.5067; BCH small-step ratio 0.5318 vs predicted 0.5; commuting BCH discrepancy 5.55e-17; commutator/BCH sweep correlation 0.9893; spectral instability observed at eta*lambda=2.01, consistent with predicted boundary 2 while eta*lambda<1 remained conservative and stable.

Limitations: Only exact 2D quadratic dynamics were tested; no MNIST/MLP, Adam/RMS/whitening implementation, noisy Hessian estimation, GPU experiment, FLOP-matched neural training, or broad hyperparameter search was performed. The controller uses oracle access to the fixed quadratic Hessian and does not implement the full neural-network integration plan.

How to run: python3 bangbang_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.11877) Sep 2, 2026

No significant difference vs. the tuned baseline (+0.4%, p=0.9286, wins 5 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 11.5879
Idea mean: 11.6306
Effect (Δ): 0.0428 (+0.4%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.9286 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±5.2%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the optimizer-matched tabular track was tested. The intervention uses elementwise Adam-like and RMS-like transformations rather than full-matrix noncommuting preconditioners, so the mechanism signature is only a behavioral direction-change proxy, not a rigorous neural BCH commutator measurement. Wall-clock/FLOP speed was not measured, and alternate static whitening/SGD comparator modes were not included.

How to run: python3 stage2_bangbang_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented and ran the Stage-2 optimizer benchmark on the registered tabular track with shared mlp_tiny architecture, tuned baseline learning rate, three parity-preserving idea settings, and 8 paired seeds. The best hard-switch result was slightly worse than baseline and highly nonsignificant, so the idea did not demonstrate a benchmark win.

Baseline: Tuned train_model baseline, lr=0.01: mean test MSE 11.587871, std 1.148191 across 8 seeds.
Idea: Hard-switch Adam-like/RMS-like optimizer, lr=0.01 and block=2: mean test MSE 11.630627, std 1.548864; paired delta +0.042755; permutation p=0.92865.

Benchmark result