# Entropy-Gap Optimizer Switch

- ID: 764
- Canonical URL: https://synthcore.org/idea/764/entropy-gap-optimizer-switch
- API JSON: https://synthcore.org/api/idea/764.json
- API Markdown: https://synthcore.org/api/idea/764.md
- Verification status: unverified
- Source: [arXiv:2607.11669](https://arxiv.org/abs/2607.11669)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: optimizer, training-dynamics, scheduler, regularization
- Math tags: statistical-mechanics, dynamical-systems, bifurcations, optimization
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Model locally competing neural-network parameter basins as low-energy states with different effective multiplicities, and inject calibrated parameter noise to measure when the optimizer begins switching between them. Use the resulting pseudo-transition peak as a principled trigger for changing learning rate, noise, or regularization rather than relying on a fixed epoch schedule.

## Mathematical statement

Consider two competing state families with energies E0 < E1, degeneracies g0 and g1, gap ΔE = E1 − E0 > 0, and inverse temperature β = 1/T. Their partition function is Z = g0 exp(−βE0) + g1 exp(−βE1). Equal statistical weight occurs at T* = ΔE / ln(g1/g0) when g1 > g0, so a small gap combined with large multiplicity generates a sharp crossover. For a neural network, replace energy by loss L, temperature by perturbation-noise scale T, and degeneracy by the empirical number or volume of perturbations mapping to a basin. If an observable takes values A0 and A1 in the two basins, its variance is Var(A) = p1(1 − p1)(A1 − A0)^2 and is maximal at p1 = 1/2. This finite variance peak is the measurable pseudo-transition signal.

## Key formulas

- $$Z=g_0e^{-\beta E_0}+g_1e^{-\beta E_1},\qquad p_1=\frac{g_1e^{-\beta E_1}}{Z}=\frac{1}{1+(g_0/g_1)e^{\beta\Delta E}}.$$
- $$T_*\simeq\frac{\Delta E}{\ln(g_1/g_0)},\qquad \operatorname{Var}(A)=p_1(1-p_1)(A_1-A_0)^2.$$
- $$\widehat{\chi}(T)=\frac{1}{M-1}\sum_{m=1}^{M}\left(A(\theta+\sqrt{2T}\,\epsilon_m)-\overline{A}_T\right)^2,\qquad \overline{A}_T=\frac{1}{M}\sum_{m=1}^{M}A(\theta+\sqrt{2T}\,\epsilon_m).$$
- $$T_{\mathrm{switch}}=\operatorname*{arg\,max}_T\widehat{\chi}(T),\qquad \eta\leftarrow\gamma_\eta\eta,\quad T\leftarrow\gamma_TT\quad\text{after the peak}.$$

## Implementation notes

1. Integration point: add a low-frequency diagnostic around an existing SGD or Adam training loop. Every K optimizer steps, freeze the current weights θ, sample M Gaussian perturbations εm with the same shape as θ, and evaluate the loss, accuracy, gradient norm, or another observable at θ + √(2T) εm for a logarithmic grid of noise temperatures T. Do not backpropagate through these probes. 2. Pseudocode: for every checkpoint and every T, compute Am = A(θ + √(2T) εm), calculate the sample variance χ[T], and select the temperature maximizing χ. Estimate a local two-basin gap by clustering probe losses into two modes; define ΔL as the difference between cluster means and estimate the multiplicity ratio using cluster counts. This gives the predicted crossover Tpred = ΔL / ln(n1/n0). If the measured peak is reproducible across adjacent checkpoints, reduce the learning rate or increase weight decay after the peak. Alternatively, retain noise if the high-loss basin still dominates. 3. Taken from the paper: the two-state partition function, entropy-gap crossover, and variance-peak criterion. Estimated empirically: basin losses, multiplicities, and response curves. 4. First cheap experiment: train a two-layer MLP on MNIST with Adam, comparing this controller against fixed cosine decay and constant Adam, using M = 32, K = 100, and eight T values. The falsifiable prediction is a finite response peak near Tpred, with measured and predicted peak temperatures agreeing within approximately a factor of two when two clusters are present. The peak should become sharper as ΔL decreases or the multiplicity ratio increases. If no reproducible peak appears across seeds, disable the controller.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.