# Double-Geometric Layerwise ES

- ID: 107
- Canonical URL: https://synthcore.org/idea/107/double-geometric-layerwise-es
- API JSON: https://synthcore.org/api/idea/107.json
- API Markdown: https://synthcore.org/api/idea/107.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.23714](https://arxiv.org/abs/2608.23714)
- Category: optimization
- Solves: accuracy, sample-efficiency, stability
- ML areas: optimizer, quantization, fine-tuning
- Math tags: optimization, statistics, probability, information-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace Gaussian perturbations in a low-dimensional neural-network optimizer with independent double-geometric integer mutations and adapt each mutation scale using its exponential-family natural gradient. Apply the method to layerwise quantization scales, adapter coefficients, pruning thresholds, or other integer/discrete hyperparameters rather than to every individual weight.

## Mathematical statement

The paper starts from the ES mutation rule \(\mathbf{x}^{(t+1)}=\mathbf{x}^{(t)}+\mathbf{z}\), where \(\mathbf{x}^{(t)}\) is the current search point and \(\mathbf{z}\) is a random mutation. For one integer coordinate, use the double-geometric distribution \(p_q(z)=\frac{1-q}{1+q}q^{|z|}\) for \(z\in\mathbb{Z}\) and \(0<q<1\). Writing \(\eta=\log q\), its sufficient statistic is \(T(z)=|z|\), and its score is \(\partial_\eta\log p_\eta(z)=|z|-\mu(q)\), where \(\mu(q)=\mathbb{E}_q|z|=\frac{2q}{1-q^2}\). For a minimization objective \(F\), the natural-gradient direction for the log-scale parameter is \(g_\eta=\operatorname{Cov}(F(\mathbf{x}+\mathbf{z}),|z|)/\operatorname{Var}(|z|)\); the covariance is estimated from sampled candidates and the denominator is the Fisher information \(\operatorname{Var}(|z|)\). A fading-memory evolution path estimates this signal online: \(h_t=\beta h_{t-1}+(1-\beta)\widehat g_t\), followed by \(\eta_{t+1}=\eta_t-\rho h_t\) for minimization, with \(q_{t+1}=\exp(\eta_{t+1})\) clipped to \([q_{\min},q_{\max}]\). For multiple coordinates or layerwise search variables, use one independent \(q_j\), one centered statistic \(|z_j|-\mu(q_j)\), and one path \(h_j\) per coordinate.

## Key formulas

- $$\mathbf{x}^{(t+1)}=\mathbf{x}^{(t)}+\mathbf{z}$$
- $$p_q(z)=\frac{1-q}{1+q}q^{|z|},\qquad z\in\mathbb{Z},\quad 0<q<1$$
- $$\frac{\partial}{\partial \eta}\log p_\eta(z)=|z|-\mu(q),\qquad \eta=\log q,\qquad \mu(q)=\mathbb{E}_q[|z|]=\frac{2q}{1-q^2}$$
- $$\widehat g_{j,t}=\frac{\sum_{k=1}^{K}u_k\left(|z_{k,j}|-\mu(q_{j,t})\right)}{\widehat{\operatorname{Var}}_{k}(|z_{k,j}|)+\varepsilon},\quad h_{j,t}=\beta h_{j,t-1}+(1-\beta)\widehat g_{j,t},\quad \eta_{j,t+1}=\operatorname{clip}(\eta_{j,t}-\rho h_{j,t},\log q_{\min},\log q_{\max})$$

## Implementation notes

Integrate this into a zero-order optimizer for a neural network whose trainable search variables are low-dimensional integer quantities: for example, one integer quantization offset or bit-allocation variable per layer, one integer rank per adapter block, or one integer perturbation coefficient per channel group. Do not mutate millions of weights initially; define a vector \(x\in\mathbb{Z}^d\) with \(d\) equal to the number of layerwise or groupwise variables, and decode \(x\) into the network before evaluating a minibatch loss or validation score. At iteration t, maintain \(q_j\) and evolution path \(h_j\) for each coordinate. Sample K mutations independently using inverse-CDF or sequential sampling from the double-geometric law: sample sign uniformly when the magnitude is nonzero, and sample magnitude m from the geometric distribution with parameter \(1-q_j\). Evaluate \(F(x+z_k)\), rank the K candidates, and assign centered utilities \(u_k\) with zero mean, such as normalized negative ranks for minimization. Compute the numerator in the displayed natural-gradient estimator using the sampled \(|z_{k,j}|\), divide by the empirical variance plus \(\varepsilon\), update the fading path with \(\beta\approx0.8\!\text{--\!}0.95\), and update \(\eta_j\) with step size \(\rho\). Clip q to prevent either no mutations or extremely heavy tails. The mathematical quantities are the double-geometric probability, analytic mean \(\mu(q)\), and Fisher normalization; utilities, empirical covariance, and q bounds are engineering choices. First test on a 2-layer MLP or small Transformer with post-training integer quantization: optimize per-layer integer zero-points and scale indices using 16--32 candidates per iteration, comparing against Gaussian ES and fixed-radius integer random search at equal validation-loss evaluations. Success is faster validation-loss reduction, fewer collapsed zero-mutation coordinates, and improved accuracy at the same quantization budget.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built Double-Geometric Layerwise ES with independent double-geometric integer mutations, analytic Fisher normalization, fading-memory natural-gradient scale adaptation, clipping, and rank utilities. The score/covariance identity matched a finite-difference derivative with absolute error 3.2e-9. On a six-coordinate layerwise-style integer quadratic using 8 fixed seeds and 2160 evaluations per run, the method reached the exact optimum in 100% of runs by iteration 20, versus 25% for Gaussian ES and 87.5% for fixed-radius integer search at the full budget; this is a promising but toy-scale signal. The learned q values reached the lower clip (0.08), so scale adaptation was effective but somewhat collapsed toward sparse mutations.

### Mechanism check

- Verdict: Built Double-Geometric Layerwise ES with independent double-geometric integer mutations, analytic Fisher normalization, fading-memory natural-gradient scale adaptation, clipping, and rank utilities. The score/covariance identity matched a finite-difference derivative with absolute error 3.2e-9. On a six-coordinate layerwise-style integer quadratic using 8 fixed seeds and 2160 evaluations per run, the method reached the exact optimum in 100% of runs by iteration 20, versus 25% for Gaussian ES and 87.5% for fixed-radius integer search at the full budget; this is a promising but toy-scale signal. The learned q values reached the lower clip (0.08), so scale adaptation was effective but somewhat collapsed toward sparse mutations.
- Confidence: 7/10
- Limitations: The mini-experiment is a low-dimensional weighted integer quadratic rather than an actual neural-network quantization or adapter task. Candidate selection is greedy-best rather than a full ES recombination/update, only 8 seeds were used, and the adaptive q values hit the lower bound; no wall-clock, GPU, quantization accuracy, or larger-dimensional scaling study was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.129
- Paired wins: 3/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/1176/bench_report.json)
- [bench_run.py](https://synthcore.org/code/1176/bench_run.py)
- [report.md](https://synthcore.org/code/1176/report.md)
- [report_bench_2026-09-03T014209.md](https://synthcore.org/code/1176/report_bench_2026-09-03T014209.md)
- [results.json](https://synthcore.org/code/1176/results.json)
- [run_experiment.py](https://synthcore.org/code/1176/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/1176)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.