# Continuation Maps for Training-Mode Transitions - ID: 2208 - Canonical URL: https://synthcore.org/idea/2208/continuation-maps-for-training-mode-transitions - API JSON: https://synthcore.org/api/idea/2208.json - API Markdown: https://synthcore.org/api/idea/2208.md - Verification status: mech_ok_no_baseline - Source: [arXiv:2608.12807](https://arxiv.org/abs/2608.12807) - Category: dynamics - Solves: stability, speedup, accuracy - ML areas: optimizer, training-dynamics, scheduler - Math tags: dynamical-systems, bifurcations, numerical-analysis, statistics - Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10 ## Idea description Treat a neural-network training run as a time-dependent dynamical system and define scalar late-time features that distinguish convergent, oscillatory, noisy, and divergent regimes. Instead of exhaustively sweeping a two-dimensional hyperparameter grid, continue the threshold curve of a feature in the learning-rate/weight-decay or learning-rate/noise plane using a secant predictor and one-dimensional correction sweep. This produces an automatically updated stability map and can be used to keep schedules on a desired side of a transition. ## Mathematical statement Let \(\theta=(\eta,\lambda)\) denote two training parameters, such as learning rate \(\eta\) and weight decay \(\lambda\), and let \(z_t(\theta)\) be a measured training trajectory after optimization step \(t\). Define a scalar feature \(F(\theta)=\Phi(z_{T-W+1:T}(\theta))\) from the final window of length \(W\), for example the coefficient of variation of gradient norms or the normalized temporal variance of validation loss. A regime boundary is the level set \(F(\theta)=c\), where \(c\) is selected from labeled stable and unstable runs. The regular-crossing condition is \(\nabla_\theta F\neq 0\); locally, the tangent direction \(\tau\) satisfies \(\nabla_\theta F\cdot\tau=0\). Given two consecutive boundary points \(\theta_{k-1}\) and \(\theta_k\), use the secant predictor \(\widehat\theta_{k+1}=\theta_k+s_k(\theta_k-\theta_{k-1})\), then hold one coordinate fixed near the predictor and solve \(F(\theta)=c\) by a bracketed one-dimensional sweep or bisection. The construction is unreliable when \(\|\nabla F\|\) is small or multiple regimes coexist within the averaging window. ## Key formulas - $$F(\eta,\lambda)=\Phi\!\left(z_{T-W+1:T}(\eta,\lambda)\right),\qquad \mathcal{B}_c=\{(\eta,\lambda):F(\eta,\lambda)=c\}.$$ - $$\nabla_\theta F(\theta)\cdot\tau(\theta)=0,\qquad \|\nabla_\theta F(\theta)\|_2>\varepsilon_F,$$ - $$\widehat\theta_{k+1}=\theta_k+s_k(\theta_k-\theta_{k-1}),\qquad F(\theta_{k+1})-c=0,$$ - $$F_{\mathrm{osc}}=\frac{\operatorname{std}_{t\in[T-W+1,T]}\ell_t}{\left|\operatorname{mean}_{t\in[T-W+1,T]}\ell_t\right|+\epsilon},\qquad F_{\mathrm{drift}}=\frac{\operatorname{std}_{t}(\|g_t\|_2)}{\operatorname{mean}_{t}(\|g_t\|_2)+\epsilon},$$ ## Implementation notes Integrate this as a diagnostic and hyperparameter-continuation tool around an existing trainer, not as a replacement for backpropagation. Choose two parameters, initially learning rate \(\eta\) and weight decay \(\lambda\), and run a short fixed-budget training trajectory from a common initialization and minibatch stream. Record loss \(\ell_t\), gradient norm \(\|g_t\|_2\), parameter displacement \(\|w_{t+1}-w_t\|_2\), and optionally validation predictions during the final window. First label a clearly stable point and a clearly oscillatory or divergent point; define a scalar feature such as \(F_{\mathrm{osc}}\) above and choose threshold \(c\) halfway between their feature values. Pseudocode: run training at \(\theta_0\), evaluate \(F(\theta_0)\); find a second point \(\theta_1\) with opposite sign of \(F-c\); predict \(\widehat\theta_2=\theta_1+s(\theta_1-\theta_0)\); fix \(\eta\) at the predicted value and bracket or bisect \(\lambda\) until \(|F-c|<\delta_F\); append the corrected point and repeat. The paper supplies the level-set, secant, and local-sweep construction; the neural-network feature, threshold, and noise-handling scheme must be estimated empirically. Start with a small MLP on MNIST or CIFAR-10, compare continuation against a dense 20-by-20 grid, and reuse checkpoints when possible. The quantitative prediction is that independently measured transition points from vertical sweeps should lie within 20 percent of the continued curve wherever \(\|\nabla F\|>\varepsilon_F\); near the curve, the sign of \(F-c\) should switch once across the normal direction. A second prediction is that tracing 20 boundary points should require substantially fewer runs than a grid while preserving the same stable or oscillatory classification. Test whether following a path with fixed margin \(F