Moment-Controlled Masked Leader Search

Integrate this as a zeroth-order optimizer over a small trainable parameter vector, not over all weights of a large model. The first target should be LoRA matrices, soft-prompt embeddings, adapter weights, or a 1,000 to 100,000 dimensional hyperparameter vector. Maintain a population X of N vectors and the best member L according to validation loss. For every candidate, sample a diagonal mask with probability p, compute Delta = L - x_i, and set t = rho / (sqrt(p) times norm(Delta) plus epsilon), so the analytical expected squared raw step is approximately rho squared. Form y_i = x_i + t times (B elementwise-multiplied by Delta), project or clip it into the allowed parameter box, evaluate the minibatch or validation objective, and retain the best N candidates plus optional elites. Pseudocode: initialize X; for each generation compute L = argmin_i f(x_i); set p = p_min + (p_max-p_min)(1-g/G); for each i sample B_j from Bernoulli(p), calculate t, propose y_i, apply repair, evaluate f(y_i), and select survivors. Estimate norms exactly; record empirical active dimensionality and coordinate coverage and compare them with d p and 1-(1-p)^K. Start with a 100M-parameter transformer, freeze the backbone, optimize rank-8 LoRA on SST-2, and compare against isotropic evolutionary perturbations and CMA-ES using the same number of forward evaluations. Success means lower validation loss or higher accuracy at equal evaluations, smoother progress, and empirical coverage close to theory; failure is no improvement or severe degradation caused by boundary repair.