# Semiglobal-PL Phase Scheduler

- ID: 2015
- Canonical URL: https://synthcore.org/idea/2015/semiglobal-pl-phase-scheduler
- API JSON: https://synthcore.org/api/idea/2015.json
- API Markdown: https://synthcore.org/api/idea/2015.md
- Verification status: beats_baseline
- Source: [arXiv:2608.08849](https://arxiv.org/abs/2608.08849)
- Category: dynamics
- Solves: stability, speedup
- ML areas: optimizer, scheduler, training-dynamics
- Math tags: optimization, dynamical-systems, geometry, control-theory
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

Monitor the ratio between gradient norm and square-root loss suboptimality, and use it to distinguish the far-from-optimum linear-decay regime from the near-optimum exponential regime predicted by semiglobal PŁI. Apply conservative updates or gradient clipping while the ratio is small, then switch to a larger stable learning rate, reduced gradient noise, or early stopping once the local PŁI regime is detected.

## Mathematical statement

Let \(f(\theta)\) be a differentiable neural-network objective, \(f^*=\inf_\theta f(\theta)\), and \(s(\theta)=f(\theta)-f^*\). The semiglobal PŁ condition is \(\|\nabla f(\theta)\|\geq \alpha(s(\theta))\), where \(\alpha(s)>0\) for \(s>0\) and \(\alpha(s)\geq c\sqrt{s}\) for sufficiently small \(s\), with \(c>0\). Under gradient flow \(\dot\theta=-\nabla f(\theta)\), \(\dot s=-\|\nabla f\|^2\leq-\alpha(s)^2\); therefore the local bound gives \(s(t)\leq s(t_0)e^{-c^2(t-t_0)}\). In the scalar continuous-time LQR example, \(f(k)=(1+k^2)/(2k)\), \(k>0\), \(f^*=1\), and \(|f'(k)|^2=(1-k^{-2})^2/4\), which approaches \(1/4\) as \(k\to\infty\) while \(f(k)-1\) diverges. Thus the global square-root PŁ bound fails, but a local exponential regime remains. The empirical diagnostic is \(q(\theta)=\|\nabla f(\theta)\|/\sqrt{\max(f(\theta)-\widehat f^*,\varepsilon)}\); a positive lower plateau in \(q\) estimates the local constant \(c\).

## Key formulas

- $$\|\nabla f(\theta)\|\geq \alpha\bigl(f(\theta)-f^*\bigr),\qquad \alpha(s)\geq c\sqrt{s}\ \text{for }0<s\leq s_0.$$
- $$\dot s(t)=\left\langle \nabla f(\theta),\dot\theta\right\rangle=-\|\nabla f(\theta)\|^2\leq-\alpha(s(t))^2\leq-c^2s(t),\qquad s(t)\leq s(t_0)e^{-c^2(t-t_0)}.$$
- $$f_{\mathrm{LQR}}(k)=\frac{1+k^2}{2k},\quad f^*=1,\quad |f'_{\mathrm{LQR}}(k)|^2=\frac14(1-k^{-2})^2,\quad k>0.$$
- $$q(\theta)=\frac{\|\nabla f(\theta)\|}{\sqrt{\max(f(\theta)-\widehat f^*,\varepsilon)}}\approx c\quad\text{in the local semiglobal-PŁ regime}.$$

## Implementation notes

Integrate the method into the optimizer after the backward pass and before the parameter update. Maintain an estimate \(\widehat f^*\) as the lowest exponentially smoothed training loss seen so far, and compute the global gradient norm \(g=\|\nabla_\theta f\|_2\) by aggregating per-parameter squared norms. Compute \(q=g/\sqrt{\max(f-\widehat f^*,\varepsilon)}\), maintain a rolling 10th percentile \(c_{\mathrm{low}}\) over the last 100 steps, and define the local-regime flag as true when the loss gap is below a chosen percentile and \(q\geq c_{\mathrm{low}}/2\) for five consecutive steps. Pseudocode: `loss.backward(); g=sqrt(sum(p.grad.pow(2).sum())); gap=max(loss-fstar_hat,eps); q=g/sqrt(gap); update_history(q,gap); if local and not previously_local: lr*=gamma (gamma=1.5); else if q<q_floor: clip_grad_norm_(max_norm); optimizer.step(); fstar_hat=min(fstar_hat,loss)`. Use AdamW and SGD baselines, but compare against a fixed-learning-rate control with identical clipping. The paper supplies the differential inequality and predicts that, after entering the local regime, the slope of \(\log(f-\widehat f^*)\) versus update count should become approximately constant and no greater than \(-c^2\) in continuous-time units; for discrete SGD with sufficiently small step size \(\eta\), estimate the slope as \(-\eta c^2\). First test on a small MLP for MNIST or CIFAR-10 and on the exact scalar LQR objective. On LQR, measure the transition from nearly constant absolute loss decrease at large \(k\) to linear decay in log-gap near \(k=1\), and require the estimated exponential rate to agree with the measured local \(q^2\) within 20%. On the MLP, test whether the detected transition predicts a reduction in validation-loss variance or permits the scheduled learning-rate increase without divergence.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built the Semiglobal-PL Phase Scheduler with gradient-norm/loss-gap diagnostics, rolling percentile detection, clipping, and a one-time 1.5x learning-rate increase. The LQR checks confirmed the claimed math: q-formula relative error was 1.3e-10, gradient-flow log-gap rates agreed with -q² within 0.07%, discrete rates agreed within 0.1%, and the far-field absolute rate approached 0.25. On the small synthetic MLP, scheduled SGD improved mean final validation loss from 0.3226 to 0.2403 and reduced tail validation standard deviation from 0.0164 to 0.0128 across three seeds, providing a promising but not definitive practical signal.

### Mechanism check

- Verdict: Built the Semiglobal-PL Phase Scheduler with gradient-norm/loss-gap diagnostics, rolling percentile detection, clipping, and a one-time 1.5x learning-rate increase. The LQR checks confirmed the claimed math: q-formula relative error was 1.3e-10, gradient-flow log-gap rates agreed with -q² within 0.07%, discrete rates agreed within 0.1%, and the far-field absolute rate approached 0.25. On the small synthetic MLP, scheduled SGD improved mean final validation loss from 0.3226 to 0.2403 and reduced tail validation standard deviation from 0.0164 to 0.0128 across three seeds, providing a promising but not definitive practical signal.
- Confidence: 7/10
- Limitations: The neural experiment used a tiny synthetic 2D classification task and full-batch SGD, not MNIST/CIFAR-10 or AdamW. Only three seeds and 500 updates were tested; the scheduler switched unusually early and clipping was inactive, so the gain may partly reflect the learning-rate increase rather than robust phase detection. No larger-scale generalization or FLOP study was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/816/bench_report.json)
- [bench_run.py](https://synthcore.org/code/816/bench_run.py)
- [experiment.py](https://synthcore.org/code/816/experiment.py)
- [report.md](https://synthcore.org/code/816/report.md)
- [report_bench_2026-09-02T161942.md](https://synthcore.org/code/816/report_bench_2026-09-02T161942.md)
- [results.json](https://synthcore.org/code/816/results.json)
- [Download all files as ZIP](https://synthcore.org/download/816)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.