# Controlled Stationary Hyperparameter Sweep

- ID: 2166
- Canonical URL: https://synthcore.org/idea/2166/controlled-stationary-hyperparameter-sweep
- API JSON: https://synthcore.org/api/idea/2166.json
- API Markdown: https://synthcore.org/api/idea/2166.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.12154](https://arxiv.org/abs/2608.12154)
- Category: sampling
- Solves: speedup, stability, sample-efficiency
- ML areas: optimizer, diffusion-sampling, training-dynamics
- Math tags: stochastic-processes, statistics, control-theory, dynamical-systems
- Ratings: usefulness 7/10; difficulty 7/10; novelty 7/10

## Idea description

Replace many independently equilibrated SGLD runs at different hyperparameters with one controlled sweep in which an auxiliary drift transports particles through the stationary distributions indexed by the swept parameter. Estimate the response of loss, predictions, uncertainty, or weight observables using covariance with the stationary generalized-potential derivative instead of finite differences between separate runs.

## Mathematical statement

Let $x$ denote a sampled state, such as network weights or a latent variable, and let $\lambda$ be a temperature, weight-decay, noise, or regularization parameter. The stationary density is $\rho_{\mathrm{st}}(x,\lambda)=\exp[-\Phi(x,\lambda)]$, where $\Phi=-\log\rho_{\mathrm{st}}$ up to normalization. For any observable $A(x,\lambda)$, the exact stationary response is $R_A(\lambda)=\partial_\lambda\langle A\rangle_\lambda=\langle\partial_\lambda A\rangle_\lambda-\operatorname{Cov}_\lambda(A,\partial_\lambda\Phi)$. During a sweep $\lambda(t)$, an auxiliary velocity $u(x,\lambda)$ must satisfy $\nabla_x\cdot[\rho_{\mathrm{st}}u]=\dot\lambda\rho_{\mathrm{st}}\partial_\lambda\Phi$, or equivalently $\nabla_x\cdot u-u\cdot\nabla_x\Phi=\dot\lambda\partial_\lambda\Phi$. This continuity equation is the mechanism that cancels distributional lag. If $\Phi$ is unknown, estimate its score $s_\psi=\nabla_x\log\rho_{\mathrm{st}}=-\nabla_x\Phi$ from samples and fit a parameterized control field using a weak residual.

## Key formulas

- $$R_A(\lambda)=\partial_\lambda\langle A\rangle_\lambda=\left\langle\partial_\lambda A\right\rangle_\lambda-\operatorname{Cov}_\lambda\left(A,\partial_\lambda\Phi\right),\qquad \Phi(x,\lambda)=-\log\rho_{\mathrm{st}}(x,\lambda).$$
- $$\nabla_x\cdot\left[\rho_{\mathrm{st}}(x,\lambda)u(x,\lambda)\right]=\dot\lambda\,\rho_{\mathrm{st}}(x,\lambda)\partial_\lambda\Phi(x,\lambda).$$
- $$\nabla_x\cdot u(x,\lambda)-u(x,\lambda)\cdot\nabla_x\Phi(x,\lambda)=\dot\lambda\,\partial_\lambda\Phi(x,\lambda).$$
- $$x_{k+1}=x_k+b_\lambda(x_k)\Delta t+\sqrt{2D\Delta t}\,\xi_k+u_\varphi(x_k,\lambda_k,\dot\lambda_k)\Delta t,\qquad \lambda_{k+1}=\lambda_k+\dot\lambda_k\Delta t,$$

## Implementation notes

Integrate the method at the sampler or optimizer level rather than in the neural-network forward pass. Represent the parameter distribution with an ensemble of particles, using network weights, a low-dimensional posterior subspace, or optimizer states as $x$. Begin with SGLD, $x\leftarrow x-\eta\nabla_xE_\lambda(x)+\sqrt{2\eta T}\xi$, and add a control network $u_\varphi(x,\lambda,\dot\lambda)$. Initialize particles at an equilibrated reference value, then sweep $\lambda$ using $\lambda_{k+1}=\lambda_k+\dot\lambda\Delta t$. At each step, compute a minibatch gradient, evaluate $u_\varphi$, update the particles, and record observables. If the energy is known, calculate $\partial_\lambda\Phi$ analytically. Otherwise estimate the score $s_\psi=\nabla_x\log\rho_{\mathrm{st}}$ from samples and train the control with random test functions $f_j$ by minimizing the weak residual $\mathbb E[\nabla f_j\cdot u_\varphi+f_j(\nabla\cdot u_\varphi+s_\psi\cdot u_\varphi-\dot\lambda\partial_\lambda\Phi)]^2$. The paper supplies the continuity equation and covariance estimator; scores, divergences, and covariances are empirically estimated. First test a two-layer MLP on MNIST or a two-dimensional toy posterior, sweeping temperature or weight decay. Compare uncontrolled SGLD, independent equilibration at each parameter, and controlled SGLD. Plot loss and predictive uncertainty versus $\lambda$, and estimate derivatives using both covariance and finite differences. The prediction is that controlled-sweep bias scales approximately as $O(\dot\lambda^2)$ near the adiabatic regime, whereas uncontrolled bias scales as $O(\tau_{\mathrm{mix}}\dot\lambda)$, with covariance derivatives agreeing with independent references within 20 percent.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built an analytic controlled SGLD sweep for the Gaussian family rho(x|a)=N(0,1/a), including covariance-response estimation, continuity-equation verification, particle sweeps, and a deterministic moment recurrence. The covariance identity matched -1/a^2 closely and the continuity residual was 1.36e-17; uncontrolled bias scaled approximately linearly with sweep rate (slope 0.90), while control reduced fastest-rate RMS error by 189x. However, the controlled deterministic error scaled with slope 0.84 rather than the predicted quadratic slope, so the required mechanism-level prediction was not confirmed under the tested discretization and worked=false.

### Mechanism check

- Verdict: Built an analytic controlled SGLD sweep for the Gaussian family rho(x|a)=N(0,1/a), including covariance-response estimation, continuity-equation verification, particle sweeps, and a deterministic moment recurrence. The covariance identity matched -1/a^2 closely and the continuity residual was 1.36e-17; uncontrolled bias scaled approximately linearly with sweep rate (slope 0.90), while control reduced fastest-rate RMS error by 189x. However, the controlled deterministic error scaled with slope 0.84 rather than the predicted quadratic slope, so the required mechanism-level prediction was not confirmed under the tested discretization and worked=false.
- Confidence: 9/10
- Limitations: Only a one-dimensional Gaussian toy family was tested, with an analytically known control field and fixed-step frozen-coefficient OU integration. No learned control network, unknown-score setting, neural-network weights, MNIST, FLOP/speed comparison, or independent-equilibration baseline was evaluated; the controlled O(v^2) asymptotic was not isolated by a timestep-convergence study.

## Artifacts

- [controlled_sweep_experiment.py](https://synthcore.org/code/868/controlled_sweep_experiment.py)
- [report.md](https://synthcore.org/code/868/report.md)
- [results.json](https://synthcore.org/code/868/results.json)
- [Download all files as ZIP](https://synthcore.org/download/868)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.