# Spectral-Width Prethermal Training Schedule

- ID: 806
- Canonical URL: https://synthcore.org/idea/806/spectral-width-prethermal-training-schedule
- API JSON: https://synthcore.org/api/idea/806.json
- API Markdown: https://synthcore.org/api/idea/806.md
- Verification status: beats_baseline
- Source: [arXiv:2607.12355](https://arxiv.org/abs/2607.12355)
- Category: training
- Solves: stability, speedup, generalization
- ML areas: training-dynamics, scheduler, optimizer, rnn, ssm
- Math tags: dynamical-systems, spectral-theory, stochastic-processes, control-theory, harmonic-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 8/10

## Idea description

Use the paper's lifetime law as a controller for training or rollout difficulty. Estimate the active perturbation bandwidth R of hidden states or forecast errors and reduce the residual gain, increase the dispersion order W, or inject controlled bandwidth whenever the estimated prethermal lifetime becomes too short.

## Mathematical statement

Let u_t denote hidden-state perturbations, gradient noise, or forecast residuals, and let P_t(k) be their normalized Fourier power spectrum. Around a selected gapless point k_0, define R_t by the second spectral moment R_t^2=sum_k d(k,k_0)^2P_t(k), where d is wrapped frequency distance. If the architecture has dispersion Delta omega(k_0+q) approximately a_W|q|^W, the paper predicts tau_t=C R_t^{-W}, where C depends on coefficient scale and noise amplitude. A target lifetime tau_star implies R_t should remain below R_max=(C/tau_star)^{1/W}. A feedback controller can update the residual gain eta using eta_{t+1}=clip(eta_t(tau_t/tau_star)^alpha,eta_min,eta_max), with alpha>0, lowering eta when the estimated lifetime is too short. Hysteresis should be used to prevent minibatch noise from causing rapid switching.

## Key formulas

- $$P_t(k)=\frac{|\widehat{u_t}(k)|^2}{\sum_{k'}|\widehat{u_t}(k')|^2},\qquad R_t^2=\sum_k d(k,k_0)^2P_t(k).$$
- $$\Delta\omega(k_0+q)\simeq a_W|q|^W\quad\Longrightarrow\quad \tau_t\simeq C R_t^{-W}.$$
- $$R_{\max}=\left(\frac{C}{\tau_\star}\right)^{1/W},\qquad \eta_{t+1}=\operatorname{clip}\left[\eta_t\left(\frac{\tau_t}{\tau_\star}\right)^\alpha,\eta_{\min},\eta_{\max}\right].$$

## Implementation notes

Integrate the controller after each optimizer step or every K training steps in a sequence model with the flat-band residual block, a state-space model, or a recurrent world model. Collect a minibatch of hidden states, residuals, or one-step forecast errors over a short temporal window, subtract the batch mean, compute an FFT along the temporal axis, and estimate P_t(k) and R_t. During a warmup phase, run fixed-bandwidth perturbation tests and fit log tau against log R to estimate C; W is computed from the architecture's Fourier symbol or estimated by a local log-log fit. During training calculate tau_hat=C max(R_t,epsilon)^{-W}. If tau_hat falls below tau_low, lower eta with the multiplicative rule, strengthen gradient clipping, or switch from a W=1 block to a preinitialized W=3 block. Restore eta only after tau_hat exceeds tau_high for several checks. Smooth R_t exponentially to avoid minibatch reactions. The first cheap experiment is a small GRU or S4-style model on adding, copy-memory, and sequential MNIST, comparing fixed learning rate, cosine decay, and this controller at equal compute. Record long-horizon loss, gradient norms, spectral width, and intervention times. The prediction is that widening the perturbation spectrum by factor b reduces retention time by approximately b^W; fitted log-time versus log-bandwidth should have slope -W, and controller interventions should cluster near the independently measured instability boundary.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built a NumPy MVP implementing wrapped FFT spectral-width estimation, the lifetime controller, smoothing, hysteresis, and reproducible toy sweeps. The mechanism predictions were confirmed: log(tau)-log(R) slopes were -1.00, -2.00, and -3.00 for W=1,2,3; bandwidth multipliers matched b^W with relative error below 5e-16; and the isolated FFT estimator had the predicted inverse-scale slope -1.00. In the noisy controller toy, feedback reduced final cumulative growth from 1.4752 to 1.1607 and reduced mean width from 0.0900 to 0.0565, although the minimum transient lifetime was not improved.

### Mechanism check

- Verdict: Built a NumPy MVP implementing wrapped FFT spectral-width estimation, the lifetime controller, smoothing, hysteresis, and reproducible toy sweeps. The mechanism predictions were confirmed: log(tau)-log(R) slopes were -1.00, -2.00, and -3.00 for W=1,2,3; bandwidth multipliers matched b^W with relative error below 5e-16; and the isolated FFT estimator had the predicted inverse-scale slope -1.00. In the noisy controller toy, feedback reduced final cumulative growth from 1.4752 to 1.1607 and reduced mean width from 0.0900 to 0.0565, although the minimum transient lifetime was not improved.
- Confidence: 8/10
- Limitations: This is an analytical toy growth model, not a trained GRU/SSM or real neural-network benchmark. The lifetime constant is known rather than fitted online, compute/FLOP overhead was not measured, and only one fixed random seed and one noisy controller trajectory were compared.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/328/bench_report.json)
- [bench_spectral_schedule.py](https://synthcore.org/code/328/bench_spectral_schedule.py)
- [report.md](https://synthcore.org/code/328/report.md)
- [report_bench_2026-08-31T011942.md](https://synthcore.org/code/328/report_bench_2026-08-31T011942.md)
- [results.json](https://synthcore.org/code/328/results.json)
- [spectral_width_experiment.py](https://synthcore.org/code/328/spectral_width_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/328)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.