Double-Geometric Layerwise ES

Integrate this into a zero-order optimizer for a neural network whose trainable search variables are low-dimensional integer quantities: for example, one integer quantization offset or bit-allocation variable per layer, one integer rank per adapter block, or one integer perturbation coefficient per channel group. Do not mutate millions of weights initially; define a vector \(x\in\mathbb{Z}^d\) with \(d\) equal to the number of layerwise or groupwise variables, and decode \(x\) into the network before evaluating a minibatch loss or validation score. At iteration t, maintain \(q_j\) and evolution path \(h_j\) for each coordinate. Sample K mutations independently using inverse-CDF or sequential sampling from the double-geometric law: sample sign uniformly when the magnitude is nonzero, and sample magnitude m from the geometric distribution with parameter \(1-q_j\). Evaluate \(F(x+z_k)\), rank the K candidates, and assign centered utilities \(u_k\) with zero mean, such as normalized negative ranks for minimization. Compute the numerator in the displayed natural-gradient estimator using the sampled \(|z_{k,j}|\), divide by the empirical variance plus \(\varepsilon\), update the fading path with \(\beta\approx0.8\!\text{--\!}0.95\), and update \(\eta_j\) with step size \(\rho\). Clip q to prevent either no mutations or extremely heavy tails. The mathematical quantities are the double-geometric probability, analytic mean \(\mu(q)\), and Fisher normalization; utilities, empirical covariance, and q bounds are engineering choices. First test on a 2-layer MLP or small Transformer with post-training integer quantization: optimize per-layer integer zero-points and scale indices using 16--32 candidates per iteration, comparing against Gaussian ES and fixed-radius integer random search at equal validation-loss evaluations. Success is faster validation-loss reduction, fewer collapsed zero-mutation coordinates, and improved accuracy at the same quantization budget.