Controlled Stationary Hyperparameter Sweep

Integrate the method at the sampler or optimizer level rather than in the neural-network forward pass. Represent the parameter distribution with an ensemble of particles, using network weights, a low-dimensional posterior subspace, or optimizer states as $x$. Begin with SGLD, $x\leftarrow x-\eta\nabla_xE_\lambda(x)+\sqrt{2\eta T}\xi$, and add a control network $u_\varphi(x,\lambda,\dot\lambda)$. Initialize particles at an equilibrated reference value, then sweep $\lambda$ using $\lambda_{k+1}=\lambda_k+\dot\lambda\Delta t$. At each step, compute a minibatch gradient, evaluate $u_\varphi$, update the particles, and record observables. If the energy is known, calculate $\partial_\lambda\Phi$ analytically. Otherwise estimate the score $s_\psi=\nabla_x\log\rho_{\mathrm{st}}$ from samples and train the control with random test functions $f_j$ by minimizing the weak residual $\mathbb E[\nabla f_j\cdot u_\varphi+f_j(\nabla\cdot u_\varphi+s_\psi\cdot u_\varphi-\dot\lambda\partial_\lambda\Phi)]^2$. The paper supplies the continuity equation and covariance estimator; scores, divergences, and covariances are empirically estimated. First test a two-layer MLP on MNIST or a two-dimensional toy posterior, sweeping temperature or weight decay. Compare uncontrolled SGLD, independent equilibration at each parameter, and controlled SGLD. Plot loss and predictive uncertainty versus $\lambda$, and estimate derivatives using both covariance and finite differences. The prediction is that controlled-sweep bias scales approximately as $O(\dot\lambda^2)$ near the adiabatic regime, whereas uncontrolled bias scales as $O(\tau_{\mathrm{mix}}\dot\lambda)$, with covariance derivatives agreeing with independent references within 20 percent.