Harvest-and-Resample Ensemble Optimizer

Integrate the mechanism as an outer loop around an existing optimizer. Maintain $N$ copies of a small MLP or policy network, with independent minibatch orderings, dropout masks, and optionally independent Adam moments. Run exactly $\tau$ optimizer updates per member, evaluate a common validation minibatch or periodically refreshed probe set, and compute score $s_i$ for every member. Pseudocode: initialize $\theta_i$; repeat: for each $i$, perform $\tau$ SGD or Adam updates; calculate $s_i$; form $S=\{i:s_i\le h\}$ for loss minimization; replace every discarded member by a uniformly sampled survivor $\theta_j$, add optional perturbation $\epsilon\sim\mathcal N(0,\sigma^2I)$, and copy or reset optimizer state; repeat. Use either a fixed survivor fraction $q$ with $h$ equal to the empirical $q$-quantile or a fixed absolute threshold. The paper supplies the propagate-then-truncate operator, not a neural-network convergence theorem; estimate the transition kernel empirically from paired pre- and post-harvest scores. First experiment: train 20 small MLPs on MNIST using SGD, comparing independent training, checkpoint averaging, and harvesting with $\tau\in\{10,100\}$ and $q\in\{0.1,0.2,0.5\}$. Start three ensembles with different initializations. Record score histograms, Wasserstein distances between ensembles, survival rates, and best and mean validation loss. The quantitative prediction is convergence toward a stationary score distribution determined mainly by $(\tau,q)$: Wasserstein distances between runs with different initial ensembles should decrease and then plateau at a common value, while changing $\tau$ or $q$ should shift the stationary histogram. In the lower-tail RL version, mean return should increase faster than unselected diffusion, with the selection advantage vanishing as $q\to1$.