Policy-Guided Terminal Trust Region for Optimizers

1. Integration point: add a short differentiable lookahead wrapper around an existing optimizer. At training step k, keep the real network parameters theta_k fixed, roll out the baseline policy pi_g for H steps to obtain c_1,...,c_H, and optimize H candidate update vectors v_0,...,v_{H-1}. Apply only v_0 to the real model, then repeat at the next minibatch. Use H=2 or 3 initially. 2. Pseudocode: compute baseline centers with c[0]=theta; for j in 0,...,H-1 set c[j+1]=c[j]+adam_update(c[j], batch_j). Initialize candidate controls from the corresponding baseline updates. Unroll theta[j+1]=theta[j]+v[j]. Evaluate J=sum(gamma**j*loss(theta[j],batch_j))+gamma**H*rho/2*norm(theta[H]-c[H])**2. Perform several inner gradient steps on v. Reject or project the candidate if norm(theta[H]-c[H]) exceeds r, and apply theta <- theta+v[0]. 3. Computed quantities are the policy rollout, terminal deviation, and objective. Estimate q by collecting pairs of nearby parameter perturbations around c_H and measuring the ratio of next-step deviations under Adam; do not assume q analytically. Choose r as the 90th percentile of baseline minibatch-induced deviations. 4. First cheap experiment: train a two-layer MLP on MNIST or CIFAR-10 with Adam as the baseline, H=2, gamma=1, and compare standard Adam, lookahead without terminal penalty, and this method at equal numbers of forward and backward evaluations. Sweep rho and r. The predicted signature is that runs with measured q<1 maintain terminal deviation below r and show geometric decay after candidate rejection or fallback, while instability or highly oscillatory loss should appear near the empirical boundary q approximately equal to 1. Test whether the measured deviation ratio after t fallback steps follows q^t within 20 percent and whether loss spikes decrease without increasing average gradient evaluations by more than 2x.