# Freeze-Out-Aware Noise Annealing

- ID: 882
- Canonical URL: https://synthcore.org/idea/882/freeze-out-aware-noise-annealing
- API JSON: https://synthcore.org/api/idea/882.json
- API Markdown: https://synthcore.org/api/idea/882.md
- Verification status: unverified
- Source: [arXiv:2607.13833](https://arxiv.org/abs/2607.13833)
- Category: training
- Solves: stability, accuracy, generalization
- ML areas: optimizer, scheduler, training-dynamics, diffusion
- Math tags: statistical-mechanics, stochastic-processes, dynamical-systems, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Replace a fixed or heuristic noise-annealing schedule with one constrained by the FPU freeze-out scaling. In stochastic gradient Langevin dynamics, reduce the injected temperature slowly enough that residual parameter fluctuations remain below a target floor; if cooling is too fast, the optimizer should retain a measurable nonequilibrium variance analogous to the FPU residual energy.

## Mathematical statement

The paper reports that an FPU system coupled to a bath cooled at rate xi falls out of equilibrium below a stochastic threshold, with residual energy E0 approximately equal to C times (xi N)^(2/3). Here N is the number of active dynamical modes, xi is minus dT/dt, T is bath temperature, and C is an empirically calibrated constant. Transfer this relation to stochastic optimization by identifying T with injected gradient-noise temperature and N with an effective number of active parameter modes. For a desired residual energy E_target, impose xi less than or equal to (E_target/C)^(3/2) divided by N. The mechanism predicts both a 2/3 residual-energy exponent and data collapse when experiments with different widths or parameter counts are plotted against xi N.

## Key formulas

- $$H_{\mathrm{FPU}}=\sum_{j=1}^{N}\frac{p_j^2}{2m}+\sum_{j=0}^{N}V(q_{j+1}-q_j)$$
- $$E_0\simeq C(\xi N)^{2/3},\qquad \xi=-\frac{\mathrm{d}T}{\mathrm{d}t}$$
- $$\xi_{\max}=\frac{1}{N}\left(\frac{E_{\mathrm{target}}}{C}\right)^{3/2}$$
- $$\theta_{k+1}=\theta_k-\eta_k\widehat{\nabla L}(\theta_k)+\sqrt{2\eta_kT_k}\,\varepsilon_k,\qquad T_{k+1}=\max(T_{\min},T_k-\eta_k\xi_k)$$

## Implementation notes

1. Integration point: apply the method to stochastic gradient Langevin dynamics, SGD with explicit Gaussian perturbations, or noisy Adam. Let theta be the parameter vector, L(theta) the loss, eta_k the step size, and T_k the noise temperature. Estimate an effective dimension N_eff as the number of parameters or principal gradient-variance modes whose running variance exceeds a threshold; initially compare this with the total parameter count. 2. Pseudocode: initialize T at T0, calibrate C, and maintain an exponential moving average theta_bar. At each step compute a normalized residual energy E_k = norm(theta_k - theta_bar_k)^2 divided by 2 tau^2. Set xi_k to the minimum of the requested cooling rate and (E_target/C)^(3/2)/N_eff. Apply theta <- theta - eta times noisy_gradient + sqrt(2 eta T) times Gaussian_noise, then set T <- max(T_min, T - eta xi_k). 3. The 2/3 exponent and xi N dependence come from the paper; C, N_eff, tau, and the mapping from parameter variance to energy must be estimated. 4. First cheap experiment: train an MLP and ResNet-18 on CIFAR-10 using constant-noise, linearly cooled, and constrained schedules, with five cooling rates and three widths. Measure terminal parameter variance after reaching T_min. The prediction is a fitted exponent near 2/3 and collapse versus xi N_eff; the adaptive schedule should keep residual energy near E_target. An exponent outside approximately 0.45 to 0.9 falsifies the direct transfer.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.