# Latent-Outbreak Learning-Rate Controller

- ID: 1405
- Canonical URL: https://synthcore.org/idea/1405/latent-outbreak-learning-rate-controller
- API JSON: https://synthcore.org/api/idea/1405.json
- API Markdown: https://synthcore.org/api/idea/1405.md
- Verification status: unverified
- Source: [arXiv:2607.25586](https://arxiv.org/abs/2607.25586)
- Category: dynamics
- Solves: stability, speedup
- ML areas: optimizer, scheduler, training-dynamics, curriculum
- Math tags: dynamical-systems, bifurcations, control-theory, stochastic-processes
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Introduce two bounded state variables into training: x measures latent, reliable learning progress, while y measures the currently active population of high-gain parameter updates or difficult examples. Let x increase irreversibly when active updates are productive, while y grows through interaction with the latent pool and decays through exhaustion. Use y to gate the learning rate or curriculum intensity, producing a low-noise incubation phase followed by an endogenous acceleration phase once x crosses the activation threshold.

## Mathematical statement

Use the three-state mean-field closure with x(t) in [0,1] denoting the fraction of parameters, examples, or representation directions that have acquired stable progress, and y(t) in [0,x(t)] denoting the fraction currently participating in high-gain updates. The latent-progress equation is \(\dot{x}=\alpha(1-x)y\), where \(\alpha>0\) is the conversion rate. A minimal expression/exhaustion equation is \(\dot{y}=\beta(x-y)y-\gamma y\), where \(\beta>0\) is the interaction-induced activation rate and \(\gamma>0\) is the exhaustion rate. Linearizing the y equation at small y gives \(\dot y\approx(\beta x-\gamma)y\), so the active-update state is locally decaying for \(x<x_c=\gamma/\beta\) and locally amplifying for \(x>x_c\). The neural controller maps y to an update-gain multiplier, for example \(\eta(t)=\eta_{\min}+y(t)(\eta_{\max}-\eta_{\min})\). The transferable property is the computable activation boundary and the predicted exponential post-threshold growth rate \(r=\beta x-\gamma\).

## Key formulas

- $$\dot{x}=\alpha(1-x)y,\qquad \dot{y}=\beta(x-y)y-\gamma y,$$
- $$\dot{y}\approx (\beta x-\gamma)y,\qquad x_c=\frac{\gamma}{\beta},$$
- $$y(t)\approx y_0\exp\!\left[(\beta x_0-\gamma)t\right]\quad\text{for nearly constant }x_0,$$
- $$\eta_t=\eta_{\min}+y_t(\eta_{\max}-\eta_{\min}),\qquad x_{t+1}=\Pi_{[0,1]}\!\left[x_t+h\alpha(1-x_t)y_t\right],\quad y_{t+1}=\Pi_{[0,x_{t+1}]}\!\left[y_t+h\{\beta(x_t-y_t)y_t-\gamma y_t\}\right].$$

## Implementation notes

1. Integration point: place the controller around SGD, Adam, or Lion. Maintain scalar states x and y per training run, layer, or parameter block. Estimate latent progress x from the fraction of examples whose prediction margin improves consistently over a window, or from the fraction of parameter blocks whose gradient cosine similarity with the previous step exceeds a threshold. Estimate active y from the fraction of blocks receiving high-confidence, high-magnitude updates; initialize y to a small positive value. Use y to interpolate between conservative and aggressive learning rates, or to control hard-example sampling. 2. Pseudocode: initialize x=x0 and y=y0. At each minibatch compute the usual gradient and a soft progress score p. Update x <- clip(x+h*alpha*(1-x)*y,0,1), optionally using an EMA of p to calibrate x. Update y <- clip(y+h*(beta*(x-y)*y-gamma*y),0,x). Set eta <- eta_min+y*(eta_max-eta_min), then apply the optimizer step with eta. Cap y if gradient norm, loss, or validation instability exceeds a safety limit. 3. Computed from the mechanism: the threshold x_c=gamma/beta and predicted growth rate beta*x-gamma. Estimated empirically: alpha, beta, and gamma from pilot runs by least-squares fitting finite differences of x and y; use transformed states if saturation causes bias. 4. First cheap experiment: train a 2-layer MLP and a small Transformer on MNIST or CIFAR-10 with fixed Adam, cosine decay, and this controller at matched average learning rate. Sweep beta/gamma so x_c is 0.3, 0.5, and 0.7. The predicted signature is a long low-y regime while x<x_c, followed by a sharp increase in log y with slope beta*x-gamma after crossing. Measured onset should occur within 20% of x_c=gamma/beta. Compare divergence rate, time-to-target-loss, early loss spikes, and final accuracy against Adam and cosine decay.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.