Symbolic Discovery of Iterative Algorithms: A Continuous Latent Space Bayesian Optimization Framework

What the math gives to ML

The paper turns the search for iterative update rules into an expected finite-horizon optimization problem, then replaces discrete symbolic-program search by Bayesian optimization in a continuous latent space learned by a variational autoencoder. The transferable asset is not a particular optimizer formula but a practical method for discovering compact, interpretable update rules against a task distribution rather than tuning a fixed hand-designed optimizer. A strong neural-network application is to search over symbolic parameter-update programs for small models, using validation loss after a short training horizon as the black-box objective and then testing the discovered rule on held-out architectures and datasets. This can produce optimizer variants that are cheaper or more stable than AdamW while retaining an auditable symbolic implementation.