# Slow-Mode-Canceling Optimizer Packet - ID: 2732 - Canonical URL: https://synthcore.org/idea/2732/slow-mode-canceling-optimizer-packet - API JSON: https://synthcore.org/api/idea/2732.json - API Markdown: https://synthcore.org/api/idea/2732.md - Verification status: unverified - Source: [arXiv:2608.25141](https://arxiv.org/abs/2608.25141) - Category: dynamics - Solves: speedup, stability - ML areas: optimizer, training-dynamics, initialization - Math tags: dynamical-systems, spectral-theory, optimization, linear-algebra - Ratings: usefulness 7/10; difficulty 6/10; novelty 8/10 ## Idea description Train two parameter replicas with symmetric coupling, treating one replica as a prepared thermalization packet for the other. Estimate the slow local Hessian direction and initialize or periodically reset the packet so that the coupled state has zero projection onto that mode; the target should then relax according to the next-slowest mode rather than the original bottleneck. ## Mathematical statement Near a stationary point, let the target and packet parameter errors be x=theta-theta^star and y=phi-theta^star, and approximate the loss gradient by grad L(theta^star+x) approximately Hx, where H is the local Hessian. Use coupled gradient-flow dynamics dot x=-Hx+kappa(y-x) and dot y=-Hy+kappa(x-y), with coupling strength kappa>0. For a Hessian eigenvector v_i satisfying Hv_i=lambda_i v_i, define scalar coordinates a_i=v_i^T x and b_i=v_i^T y. Their common and difference coordinates are c_i=(a_i+b_i)/2 and d_i=(a_i-b_i)/2, obeying dot c_i=-lambda_i c_i and dot d_i=-(lambda_i+2kappa)d_i. If lambda_10.$$ - $$Hv_i=\lambda_i v_i,\quad c_i=\frac{v_i^\top x+v_i^\top y}{2},\quad d_i=\frac{v_i^\top x-v_i^\top y}{2},\qquad \dot c_i=-\lambda_i c_i,\quad \dot d_i=-(\lambda_i+2\kappa)d_i.$$ - $$v_1^\top y(0)=-v_1^\top x(0)\ \Rightarrow\ c_1(0)=0,\qquad r_{\mathrm{after}}\approx\min\{\lambda_2,\lambda_1+2\kappa\}.$$ - $$\theta_{t+1}=\theta_t-\eta[g_\theta+\kappa(\theta_t-\phi_t)],\qquad \phi_{t+1}=\phi_t-\eta[g_\phi+\kappa(\phi_t-\theta_t)].$$ ## Implementation notes 1. Integration point: add a second trainable parameter vector phi with the same architecture as the ordinary model parameter vector theta. At every minibatch, compute both losses and update both replicas with gradients plus a parameter-coupling term. Use the target prediction from theta; use phi only as the packet and optionally average predictions late in training. The discrete Euler updates are theta <- theta-eta[g_theta+kappa(theta-phi)] and phi <- phi-eta[g_phi+kappa(phi-theta)]. 2. Pseudocode: initialize theta and phi, estimate a low-curvature direction v1 using 5--20 Hessian-vector products and Lanczos or inverse iteration, then estimate the displacement x relative to a nearby reference theta_star or an initial checkpoint. Set phi so that v1^T(phi-theta_star)=-v1^T(theta-theta_star), while leaving orthogonal components unchanged. Run the coupled updates and re-estimate v1 every 100--1000 steps. 3. Computed versus estimated: the modal equations and cancellation condition are analytic; v1, lambda_1, and lambda_2 are estimated from Hessian-vector products or by fitting exponential decay rates of projected errors. Minibatch noise will continually repopulate the cancelled mode, so measure its replenishment rate. 4. First cheap experiment: use a two-layer MLP on MNIST and compare SGD, an unprepared two-replica optimizer, and the prepared optimizer at matched total parameter-update FLOPs. Measure loss, validation accuracy, and error projected onto v1. The falsifiable prediction is that the late-time projected error changes from A exp(-lambda_1 t) to approximately B exp(-min(lambda_2,lambda_1+2kappa)t). Increasing kappa should improve the decay rate only until lambda_1+2kappa approximately equals lambda_2. A cancellation ratio |c_1(0)|/|a_1(0)| below 0.1 should yield at least a tenfold reduction in slow-mode amplitude. For discrete updates, instability should appear when eta(lambda_max+2kappa) reaches approximately 2. ## Verification - Status: unverified - Mechanism evidence: yes - Mechanism confirmed: no - Verdict: Built and numerically verified the coupled two-replica optimizer on a diagonal quadratic Hessian. Exact slow-mode cancellation was observed, the measured rate matched the discrete prediction across the kappa sweep to numerical precision, saturation occurred at kappa=0.3 as predicted, and the Euler stability boundary was predicted at kappa=19.0 with divergence observed at kappa=19.05. The prepared packet reached 1.05% of the unprepared final loss in this toy case. ### Mechanism check - Verdict: Built and numerically verified the coupled two-replica optimizer on a diagonal quadratic Hessian. Exact slow-mode cancellation was observed, the measured rate matched the discrete prediction across the kappa sweep to numerical precision, saturation occurred at kappa=0.3 as predicted, and the Euler stability boundary was predicted at kappa=19.0 with divergence observed at kappa=19.05. The prepared packet reached 1.05% of the unprepared final loss in this toy case. - Confidence: 9/10 - Limitations: Only a deterministic diagonal quadratic/local-Hessian model was tested; no MLP, MNIST, stochastic minibatch replenishment, Hessian-vector estimation, nonlinear effects, wall-clock or FLOP comparison, or practical packet-reset schedule was evaluated. ## Artifacts - [bench_experiment.py](https://synthcore.org/code/1081/bench_experiment.py) - [bench_report.json](https://synthcore.org/code/1081/bench_report.json) - [report.md](https://synthcore.org/code/1081/report.md) - [results.json](https://synthcore.org/code/1081/results.json) - [slow_mode_experiment.py](https://synthcore.org/code/1081/slow_mode_experiment.py) - [Download all files as ZIP](https://synthcore.org/download/1081) ## Disclaimer AI-generated research hypothesis, automatically tested. Not peer-reviewed.