Bound-Optimized Stochastic Preconditioner

Use this as an outer optimizer around a small neural network or around a low-dimensional block of a larger model. Parameterize each candidate task as \(M_k=L_kL_k^T+\epsilon I\), using diagonal, block-diagonal, or low-rank \(L_k\) so candidate covariance square roots are inexpensive. During an adaptation window of \(H\) optimizer steps, generate \(N\) perturbed parameter trajectories under a baseline task \(M_0\), using the same minibatch sequence and Gaussian random seeds for all candidate comparisons. Define \(S(\tau_i)\) as accumulated training loss plus \(\alpha\sum_t\|u_{i,t}\|_{R}^2\). For every candidate, compute the Gaussian trajectory log-density ratio, subtract the maximum log weight for numerical stability, normalize the weights, calculate \(\widehat J(M_k)\), and reject candidates with \(\operatorname{ESS}(M_k)<\rho N\), initially \(\rho=0.2\). Select the feasible minimum and use its task for the next adaptation window. The control update can be the weighted perturbation average \(u_t\leftarrow\sum_i\bar w_i u_{i,t}\). The paper supplies the coupling relation, change-of-measure objective, and upper-bound rationale; the neural experiment must estimate trajectory costs, density ratios, bound tightness, and covariance stability. First test a two-layer MLP on MNIST with SGD, Adam, and this method, using \(H=10\), \(N=64\), and diagonal covariance scales from \(10^{-3}\) to \(10^1\). Compare shared-sample candidate rankings against fresh-sample rankings. The predicted signature is Spearman ranking correlation above 0.8 in the adequate-ESS regime, followed by a sharp ranking degradation when ESS drops below 20 percent of samples. Also plot loss variance against the largest eigenvalue of \(M\); instability and failed runs should increase sharply beyond a task-dependent boundary, while BOTC should select below that boundary.