△ Mechanism confirmed, baseline not beaten 2026

Continuation Maps for Training-Mode Transitions

Implementation & benchmark of arXiv:2608.12807 — Feature-Based Continuation of Pattern Transitions in a One-Dimensional Brusselator

Usefulness7/10

Difficulty4/10

Novelty7/10

Dynamics Stability Speedup Accuracy Optimizer Training dynamics Scheduler Dynamical systems Bifurcations Numerical analysis Statistics

Source paper: Feature-Based Continuation of Pattern Transitions in a One-Dimensional Brusselator arXiv:2608.12807 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat a neural-network training run as a time-dependent dynamical system and define scalar late-time features that distinguish convergent, oscillatory, noisy, and divergent regimes. Instead of exhaustively sweeping a two-dimensional hyperparameter grid, continue the threshold curve of a feature in the learning-rate/weight-decay or learning-rate/noise plane using a secant predictor and one-dimensional correction sweep. This produces an automatically updated stability map and can be used to keep schedules on a desired side of a transition.

Formulas

$$F(\eta,\lambda)=\Phi\!\left(z_{T-W+1:T}(\eta,\lambda)\right),\qquad \mathcal{B}_c=\{(\eta,\lambda):F(\eta,\lambda)=c\}.$$

$$\nabla_\theta F(\theta)\cdot\tau(\theta)=0,\qquad \|\nabla_\theta F(\theta)\|_2>\varepsilon_F,$$

$$\widehat\theta_{k+1}=\theta_k+s_k(\theta_k-\theta_{k-1}),\qquad F(\theta_{k+1})-c=0,$$

$$F_{\mathrm{osc}}=\frac{\operatorname{std}_{t\in[T-W+1,T]}\ell_t}{\left|\operatorname{mean}_{t\in[T-W+1,T]}\ell_t\right|+\epsilon},\qquad F_{\mathrm{drift}}=\frac{\operatorname{std}_{t}(\|g_t\|_2)}{\operatorname{mean}_{t}(\|g_t\|_2)+\epsilon},$$

Mathematical statement

Let $\theta=(\eta,\lambda)$ denote two training parameters, such as learning rate $\eta$ and weight decay $\lambda$, and let $z_t(\theta)$ be a measured training trajectory after optimization step $t$. Define a scalar feature $F(\theta)=\Phi(z_{T-W+1:T}(\theta))$ from the final window of length $W$, for example the coefficient of variation of gradient norms or the normalized temporal variance of validation loss. A regime boundary is the level set $F(\theta)=c$, where $c$ is selected from labeled stable and unstable runs. The regular-crossing condition is $\nabla_\theta F\neq 0$; locally, the tangent direction $\tau$ satisfies $\nabla_\theta F\cdot\tau=0$. Given two consecutive boundary points $\theta_{k-1}$ and $\theta_k$, use the secant predictor $\widehat\theta_{k+1}=\theta_k+s_k(\theta_k-\theta_{k-1})$, then hold one coordinate fixed near the predictor and solve $F(\theta)=c$ by a bracketed one-dimensional sweep or bisection. The construction is unreliable when $\|\nabla F\|$ is small or multiple regimes coexist within the averaging window.

Implementation notes

Integrate this as a diagnostic and hyperparameter-continuation tool around an existing trainer, not as a replacement for backpropagation. Choose two parameters, initially learning rate $\eta$ and weight decay $\lambda$, and run a short fixed-budget training trajectory from a common initialization and minibatch stream. Record loss $\ell_t$, gradient norm $\|g_t\|_2$, parameter displacement $\|w_{t+1}-w_t\|_2$, and optionally validation predictions during the final window. First label a clearly stable point and a clearly oscillatory or divergent point; define a scalar feature such as $F_{\mathrm{osc}}$ above and choose threshold $c$ halfway between their feature values. Pseudocode: run training at $\theta_0$, evaluate $F(\theta_0)$; find a second point $\theta_1$ with opposite sign of $F-c$; predict $\widehat\theta_2=\theta_1+s(\theta_1-\theta_0)$; fix $\eta$ at the predicted value and bracket or bisect $\lambda$ until $|F-c|<\delta_F$; append the corrected point and repeat. The paper supplies the level-set, secant, and local-sweep construction; the neural-network feature, threshold, and noise-handling scheme must be estimated empirically. Start with a small MLP on MNIST or CIFAR-10, compare continuation against a dense 20-by-20 grid, and reuse checkpoints when possible. The quantitative prediction is that independently measured transition points from vertical sweeps should lie within 20 percent of the continued curve wherever $\|\nabla F\|>\varepsilon_F$; near the curve, the sign of $F-c$ should switch once across the normal direction. A second prediction is that tracing 20 boundary points should require substantially fewer runs than a grid while preserving the same stable or oscillatory classification. Test whether following a path with fixed margin $F<c$ prevents late-training loss oscillations and whether the boundary bends systematically as batch size or gradient noise changes.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified a continuation-map MVP for a diagonal quadratic training dynamical system. The measured instability boundary matched eta*(4+lambda)=2 with 0.054% mean relative error, the local normal sweep switched feature sign, and secant prediction plus scalar bisection tracked the boundary with 1.13e-7 mean relative error. Continuation used 16 trajectory evaluations versus 400 for the 20x20 grid, confirming the mechanism in this toy system but not yet in a neural-network trainer.

Agent confidence: 9/10

Baseline: 20x20 dense grid: 400 runs, mean relative boundary error 0.04044
Idea: Secant continuation plus 1D bisection: 16 runs, mean relative boundary error 1.13e-7; direct sweep mean error 5.42e-4; scaling mean eta_boundary*(4+lambda)=2.00108

Limitations: Only a deterministic 2D diagonal quadratic was tested; no MLP, minibatch noise, validation loss, real optimizer, checkpoint reuse, schedule-following test, or noisy/multiregime boundary was evaluated. The toy feature is asymptotic log-norm growth rather than the exact proposed loss coefficient of variation.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.12807) Sep 2, 2026

Significantly WORSE than the tuned baseline (n/a, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0
Idea mean: 0.2629
Effect (Δ): 0.26 (n/a; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 0 configs

Limitations:

Only the built-in dynamics track was tested because stability/control is structurally matched there. The correction was a short noisy local controller rather than a full multi-point boundary map with checkpoint reuse; schedule-following and batch-size/noise continuation studies were not run.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented continuation-based late-window oscillation control on the structurally matched dynamics track using the shared rnn_small architecture, equal learning-rate sweeps, and 8 paired seeds. The mechanism signature was confirmed: mean oscillation feature fell from 0.5832 to 0.1645, a 71.8% reduction, but test MSE worsened significantly from 0.002912 to 0.262946 (paired delta +0.260034, permutation p=0.0081). The diagnostic effect is real, but the proposed intervention did not improve the standard task metric.

Baseline: Best fixed-SGD lr=0.01; test MSE mean 0.0029119684, std 0.0019628704
Comparison verdict: idea worse (significant)
Idea: Best continuation lr=0.01; test MSE mean 0.2629458737, std 0.0624234057; oscillation feature 0.1644519 versus baseline 0.5831796; relative reduction 71.80%; final lr mean 0.00029296875
Paired delta: 0.26003390522964764
Permutation p value: 0.0081

Benchmark result