# Singular-perturbation continuation schedule

- ID: 972
- Canonical URL: https://synthcore.org/idea/972/singular-perturbation-continuation-schedule
- API JSON: https://synthcore.org/api/idea/972.json
- API Markdown: https://synthcore.org/api/idea/972.md
- Verification status: unverified
- Source: [arXiv:2607.15692](https://arxiv.org/abs/2607.15692)
- Category: training
- Solves: stability, speed
- ML areas: optimizer, training-dynamics, rnn, ssm
- Math tags: dynamical-systems, bifurcations, linear-algebra, numerical-analysis
- Ratings: usefulness 6/10; difficulty 6/10; novelty 8/10

## Idea description

Use the paper's fast-layer/reduced-problem decomposition as a training schedule: first optimize a cheap reduced neural dynamics on the critical manifold, then gradually restore the fast dynamics by increasing the stiffness parameter. This provides a continuation path from an easy slow problem to the intended recurrent or implicit model and supplies a concrete stopping criterion based on normal-hyperbolicity loss.

## Mathematical statement

For the paper's system, the singular limit \(\varepsilon\to0\) separates a fast layer, in which P,S are constant, from a reduced slow problem on \(\mathcal C_0\). The reduced equations include \(P'=S\) and \(S'=F(P,T)\), while the critical-manifold constraint determines T from P. The same construction can initialize a neural dynamical system with an algebraic state \(T=h_\theta(P)\), train the reduced map first, and then use finite \(\varepsilon\) dynamics \(\varepsilon \dot z=f_\theta(z,h_\theta(P))\), \(\dot T=U\), \(\dot U=G_\theta(P,T)\). The continuation is safe only while the transverse fast Jacobian has negative real parts bounded away from zero; define \(\mu=-\max\operatorname{Re}\lambda(J_\perp)\), and stop increasing stiffness when \(\mu\leq\mu_{\min}\).

## Key formulas

- $$P'=S,\qquad S'=\frac{P}{1-\rho\phi(T)}\left[d-g\left(1-\frac{P}{1-\rho\phi(T)}\right)\right],\qquad P=T(1-\rho\phi(T)).$$
- $$\varepsilon\dot z=f_\theta(z,T,P),\qquad \dot T=U,\qquad \dot U=G_\theta(P,T).$$
- $$\mu(P,T)=-\max_{\lambda\in\operatorname{eig}(J_\perp(P,T))}\operatorname{Re}\lambda,\qquad J_\perp=\frac{\partial f_\theta}{\partial z};\quad \mu>\mu_{\min}>0\text{ is the continuation condition}.$$
- $$\varepsilon_{r+1}=\min\left(\varepsilon_{\max},\;\varepsilon_r\exp\left[\kappa\,\operatorname{clip}\left(\frac{\mu}{\mu_{\min}}-1,-1,1\right)\right]\right),\quad \kappa>0.$$

## Implementation notes

1. Integration point: apply this to an implicit recurrent layer or neural ODE whose hidden state is split into slow memory P and fast auxiliary state z. During stage zero, replace the fast state by a learned algebraic decoder z=h_theta(P), and train the reduced model with ordinary backpropagation. 2. Pseudocode: initialize epsilon=1; train the reduced loss using z=h_theta(P); construct the finite-epsilon cell; for each continuation stage, run a short warm start, estimate the largest real eigenvalue of J_perp using 5-10 power iterations or autograd Jacobian-vector products, and decrease epsilon by a factor 0.5 only if mu>mu_min; otherwise reduce learning rate, add a barrier -lambda log(mu), or stop continuation. Train to convergence at each stage. 3. Computed from the paper: the fast/reduced decomposition and the requirement that trajectories remain near a normally hyperbolic critical manifold. Estimated: J_perp, mu, reconstruction error ||z-h_theta(P)||, and task loss. 4. First cheap experiment: use a 1-2 layer implicit MLP on a synthetic long-horizon nonlinear recurrence with known stiff and nonstiff regimes, comparing direct training against continuation. Measure fixed-point failures and gradient norms as epsilon decreases. The predicted signature is a sharp rise in solver iterations and gradient variance when mu approaches zero; successful continuation should maintain mu>mu_min and reduce final training failures, while reduced-to-full trajectory discrepancy should initially scale approximately linearly with epsilon away from folds.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.