# Critical-Rate Learning-Rate Controller

- ID: 678
- Canonical URL: https://synthcore.org/idea/678/critical-rate-learning-rate-controller
- API JSON: https://synthcore.org/api/idea/678.json
- API Markdown: https://synthcore.org/api/idea/678.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.10217](https://arxiv.org/abs/2607.10217)
- Category: dynamics
- Solves: stability, speed
- ML areas: optimizer, scheduler, training-dynamics
- Math tags: dynamical-systems, bifurcations, control-theory, optimization, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace a fixed or manually scheduled learning rate with a feedback controller that estimates the critical rate of a saddle-node-like training mode and slows the schedule before the mode overshoots. The controller is applied to a low-dimensional observable of training, while ordinary gradient updates remain unchanged. It should permit aggressive learning-rate increases away from the bifurcation and automatically reduce them near a sharp stability boundary.

## Mathematical statement

Use the nonautonomous saddle-node normal form \(\dot{x}=x^{2}+u(t)\), where \(x(t)\) is a scalar training-mode coordinate and \(u(t)\) is a controlled schedule parameter. For fixed \(u<0\), the stable and unstable equilibria are \(x_s(u)=-\sqrt{-u}\) and \(x_u(u)=\sqrt{-u}\), with stable linearization \(\partial_x(x^2+u)|_{x_s}=-2\sqrt{-u}\). A moving stable branch has velocity \(\dot{x}_s=\dot{u}/(2\sqrt{-u})\). The local tracking-lag approximation is \(\delta\approx \dot{x}_s/(2\sqrt{-u})=\dot{u}/(4(-u))\). Requiring this lag to remain smaller than the distance from the stable to unstable branch, \(\sqrt{-u}\), gives the approximate critical-rate boundary \(|\dot{u}|<r_c(u)=4(-u)^{3/2}\). The paper's central mechanism is that the exact critical rate is unique for each forcing trajectory; in the neural implementation, the exact rate function is estimated by short rollouts and the analytic expression above is used as a cheap prior. The controller minimizes task loss plus a penalty for violating the rate boundary.

## Key formulas

- $$\dot{x}=x^{2}+u(t),\qquad x_s(u)=-\sqrt{-u},\quad x_u(u)=\sqrt{-u},\quad u<0.$$
- $$\lambda_s(u)=\partial_x(x^2+u)|_{x=x_s}=-2\sqrt{-u},\qquad \dot{x}_s(u)=\frac{\dot{u}}{2\sqrt{-u}}.$$
- $$|\dot{u}(t)|<r_c(u(t)),\qquad r_c(u)\approx4(-u)^{3/2},$$
- $$J[u]=\mathcal{L}_{\mathrm{train}}(u)+\beta\int_0^T\left[\max\left(0,\frac{|\dot{u}(t)|}{\widehat r_c(u(t))+\epsilon}-1\right)\right]^2dt.$$

## Implementation notes

1. Integration point: use this as an outer controller for the learning-rate or optimizer-gain schedule. At every optimizer step, retain a scalar observable x_hat measuring the potentially unstable training mode. A cheap choice is the projection of the parameter update onto the previous update, x_hat_t = clip((Delta theta_t dot Delta theta_{t-1})/(||Delta theta_{t-1}||^2+epsilon), -M, M); alternatively use the largest Ritz eigenvalue of the minibatch Hessian or the norm of a recurrent hidden state. Let u_t be a normalized schedule variable controlling learning rate eta_t = eta_min + sigmoid(u_t)(eta_max-eta_min). 2. Pseudocode: initialize u and an exponential moving estimate r_hat. For each step, compute the normal gradient update with eta(u); estimate x_hat and its one-step drift v_hat = (x_hat_t-x_hat_{t-1})/Delta_t; estimate local distance d_hat from the tracked branch, or use d_hat = sqrt(max(-u_t, epsilon)); compute r_prior = 4*max(-u_t, epsilon)^(3/2). Set r_hat = alpha*r_hat + (1-alpha)*abs(v_hat). If abs(delta_u/Delta_t) > gamma*r_prior or x_hat approaches the estimated unstable side, replace delta_u by sign(delta_u)*gamma*r_prior*Delta_t; otherwise keep the proposed schedule update. 3. The saddle-node equilibria, eigenvalue, and 4(-u)^(3/2) scaling are analytic. The mapping from network statistics to x_hat and the exact critical-rate function are empirical and should be estimated by short perturbation rollouts: temporarily increase the schedule rate by factors 1.0, 1.25, 1.5, and record whether x_hat returns to the tracked branch or overshoots. 4. First cheap experiment: train a small two-layer MLP on MNIST or CIFAR-10 with SGD, cosine warmup, and the controller versus the same schedule without control. Sweep warmup slopes over a logarithmic grid. Plot update norm, loss, x_hat, and schedule slope. The quantitative prediction is a sharp overshoot boundary: after normalizing by r_prior, runs with slope ratio below approximately 1 should remain tracking, while ratios above 1 should show a rapid increase in update norm or loss variance; the measured transition should fall within 20% of the predicted ratio. A secondary prediction is that adaptive control reduces overshoot events without reducing the maximum stable average learning rate.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a self-contained critical-rate controller and toy verification in critical_rate_experiment.py. The analytic predictions for r_c scaling and moving-branch velocity were confirmed: observed exponent 1.5000000000000002 versus 1.5, with mean relative velocity error 7.8e-9; controller clipping also passed all tests. However, the predicted normalized transition near q=1 was not observed: no branch crossing occurred even up to q=100, so the central mechanism was not numerically confirmed. The quadratic experiment showed less divergence with control, but at a substantially lower maximum learning rate, so it does not establish the promised effect.

### Mechanism check

- Verdict: Built a self-contained critical-rate controller and toy verification in critical_rate_experiment.py. The analytic predictions for r_c scaling and moving-branch velocity were confirmed: observed exponent 1.5000000000000002 versus 1.5, with mean relative velocity error 7.8e-9; controller clipping also passed all tests. However, the predicted normalized transition near q=1 was not observed: no branch crossing occurred even up to q=100, so the central mechanism was not numerically confirmed. The quadratic experiment showed less divergence with control, but at a substantially lower maximum learning rate, so it does not establish the promised effect.
- Confidence: 9/10
- Limitations: No MNIST/MLP experiment was run. The controller used only the analytic prior and did not implement empirical rollout estimation of the exact critical rate or a loss penalty. The nonlinear sweep used a linear ramp ending just before u=0 and therefore cannot assess post-bifurcation behavior; the baseline comparison is a small noisy quadratic with a more aggressive schedule than the controlled run.

## Artifacts

- [critical_rate_experiment.py](https://synthcore.org/code/297/critical_rate_experiment.py)
- [report.md](https://synthcore.org/code/297/report.md)
- [results.json](https://synthcore.org/code/297/results.json)
- [Download all files as ZIP](https://synthcore.org/download/297)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.