Fleming-Viot Stable-Trajectory Optimizer

Integrate the method as an outer wrapper around SGD or Adam. Maintain N parameter replicas, each with its own optimizer state, minibatch stream, and age. At every step, compute a minibatch gradient and update each nonfailed replica. Define the absorbing set B using one or more fixed conditions: validation loss above a threshold, gradient norm above a threshold, NaN or Inf detection, parameter norm outside a trust region, or a sudden loss increase. After updates, mark failed replicas. For each failed index i, sample a surviving index j uniformly, copy both parameters and optimizer state from j to i, and set age[i] = 0; increment the ages of all other replicas. A constant hazard can be implemented as failure probability p per step; an age-dependent hazard uses q(a) = 1 - exp(-mu(a) Delta t). The paper provides the age-space construction, hazard survival relation, cloning mechanism, and conditional exponential convergence; estimate the decay rate and finite-particle exponent from experiments. First test: train a 32-replica MLP on MNIST with an aggressive learning rate, comparing ordinary SGD, independent replicas, and Fleming–Viot SGD at equal total gradient budget. Record survivor fraction, loss, and parameter diversity. The prediction is a sharp stability transition as eta increases: ordinary SGD should diverge above a critical eta, while the particle system remains populated until replacement frequency becomes excessive. Fit log survival probability versus time and test whether the late-time conditional distribution approaches a stationary ensemble with residual slope approximately -r. Repeat with N = 8, 16, 32, 64 and check polynomial error reduction relative to an N = 256 reference ensemble.