Latent Bayesian Discovery of Symbolic Optimizers

(1) Integration point: replace the optimizer-update block in a small PyTorch training loop. Represent each candidate as a fixed-length program with tokens such as grad, momentum, square, sqrt, add, mul, div, neg, scalar constants, and state assignments. Restrict the grammar so every program returns an update tensor and uses numerically safe operations; clamp denominators by $\epsilon=10^{-8}$, bound scalar constants, and reject programs with undefined state references. The program receives parameter $\theta_t$, gradient $g_t$, state tensors, and step index $t$, and returns $\Delta_t$ and next state, with $\theta_{t+1}=\theta_t+\Delta_t$. (2) Pseudocode: sample an initial library of valid token programs $z_i$; train the VAE on these programs using the stated reconstruction-plus-KL loss; encode each program to $h_i=\mu_\varphi(z_i)$; run each optimizer for $T$ steps on several training tasks and compute $\hat J(z_i)=M^{-1}\sum_m\ell^{val}_{D_m}(\theta_T^{(m)})$; fit a GP to $(h_i,\hat J_i)$; select $h_{new}=\arg\min_h[\mu(h)-\kappa\sigma(h)]$; decode $h_{new}$ to a token program, repair invalid syntax, evaluate it, append the result, and refit the GP. Repeat for 50--200 proposals, then retrain the best programs from scratch. (3) Computed from the paper's machinery: the finite-horizon expected objective, VAE latent encoding, and GP acquisition. Estimated empirically: validation scores, GP hyperparameters, and the task distribution; use 8--16 random seeds and 4--8 small tasks per candidate to reduce noisy rankings. (4) First experiment: use CIFAR-10 with a 2-layer MLP and a tiny ResNet, compare against SGD, momentum SGD, Adam, and AdamW under equal 100-step rollout budgets. Search first with width 256 and 32 channels, then evaluate the best symbolic rules on larger widths and unseen seeds. Success means lower median validation loss at 100 and 500 steps, fewer NaN or divergence events, and competitive final accuracy; report discovery compute separately and require the discovered rule to beat AdamW on held-out tasks rather than only on search rollouts.