Leader-Directed Differential Evolution for Adapter Training

Integrate this at the optimizer level for a frozen pretrained network with trainable LoRA, prefix-tuning, or soft-prompt parameters. Flatten each candidate's trainable parameters into x_i, while retaining a layer map so the differential vector can optionally be normalized separately per tensor. For layer l, compute d_l = (x_r1,l - x_r2,l)/(RMS(x_r1,l-x_r2,l)+epsilon), and use the corresponding normalized leader difference if per-layer scale control is enabled. At every generation, identify x_best using a fixed validation minibatch or an averaged score over two minibatches. For each target, sample distinct r1 and r2, sample C_j from Bernoulli(CR), force one random coordinate or tensor block to be active, calculate u_i = x_i + C elementwise-multiplied by [F(x_best-x_i) + F(x_r1-x_r2)], project into LoRA norm bounds, evaluate the candidate, and retain elites plus the best offspring. Use population size 16 to 32, F in [0.3,0.8], and CR in [0.5,0.9] for the first sweep. Record raw proposal norms, changed-coordinate fraction, coordinate coverage, repaired-coordinate fraction, and loss improvement per forward evaluation. The first experiment should use a small ViT or 100M-parameter transformer with rank-4 or rank-8 adapters on CIFAR-100 or SST-2, comparing against random search, isotropic evolutionary strategies, and gradient-based LoRA under an equal forward-pass budget. The predicted signal is faster validation improvement than isotropic perturbations when population differences capture useful adapter directions; a negative result is equal or worse loss descent after accounting for population evaluations.