# Bounded predictive-gain optimizer

- ID: 2037
- Canonical URL: https://synthcore.org/idea/2037/bounded-predictive-gain-optimizer
- API JSON: https://synthcore.org/api/idea/2037.json
- API Markdown: https://synthcore.org/api/idea/2037.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.09218](https://arxiv.org/abs/2608.09218)
- Category: optimization
- Solves: speedup, stability, generalization
- ML areas: optimizer, training-dynamics, fine-tuning
- Math tags: optimization, convex-analysis, information-theory, dynamical-systems
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Replace a fixed learning rate for each layer or parameter block with a bounded gain selected by the one-step-ahead predictive loss. The sign of the product between the current gradient and the next gradient estimates whether the previous update moved downhill: aligned gradients increase the gain, while sign reversals decrease it. A mirror-descent update on a bounded interval prevents the runaway step sizes that can occur with exponential or unconstrained learning-rate parameterizations.

## Mathematical statement

The paper models a conditional predictive family as $P_{\lambda}(\mathrm{d}y)=p(y\mid\lambda)\nu(\mathrm{d}y)$, where $\lambda\in\Lambda\subset\mathbb{R}^{m}$ is the state and $\nu$ is a dominating measure. A score-driven state update has the generic form $\lambda_{t+1}=\omega(\lambda_t)+G_t s_t$, where $s_t=\nabla_{\lambda}\log p(y_t\mid\lambda_t)$ is the score and $G_t$ is a gain. The paper's central observation is that the negative product of consecutive scores is the stochastic gradient of a one-step predictive loss with respect to the gain. For neural-network loss $f_t(\theta)$, let $g_t=\nabla_{\theta}f_t(\theta_t)$ and update $\theta_{t+1}=\theta_t-a_t\odot g_t$. The gain derivative is estimated by $\widehat{\nabla}_{a_t}\ell_t=-g_{t+1}\odot g_t$, because increasing $a_{t,i}$ changes the next parameter by $-g_{t,i}$ and the observed next loss gradient is $g_{t+1,i}$. Thus aligned consecutive gradients increase the gain and gradient reversals decrease it. Let $a_i\in[a_{\min},a_{\max}]$, let $\psi$ be differentiable and strictly convex, and define $D_{\psi}(x,y)=\psi(x)-\psi(y)-\langle\nabla\psi(y),x-y\rangle$. With reference gain $a^{\mathrm{ref}}$ and persistence coefficient $\rho\ge0$, the mirror update is $a_{t+1}=\arg\min_{a\in[a_{\min},a_{\max}]^d}\{\eta\langle\widehat{\nabla}_{a_t}\ell_t,a\rangle+D_{\psi}(a,a_t)+\rho D_{\psi}(a,a^{\mathrm{ref}})\}$. Here $d$ is the number of gain groups and $\eta$ is the gain learning rate.

## Key formulas

- $$P_{\lambda}(\mathrm{d}y)=p(y\mid\lambda)\nu(\mathrm{d}y),\qquad \lambda\in\Lambda\subset\mathbb{R}^{m}.$$
- $$\theta_{t+1}=\theta_t-a_t\odot g_t,\qquad g_t=\nabla_{\theta}f_t(\theta_t),$$
- $$\widehat{\nabla}_{a_t}\ell_t=-g_{t+1}\odot g_t,$$
- $$a_{t+1}=\arg\min_{a\in[a_{\min},a_{\max}]^d}\left\{\eta\left\langle-g_{t+1}\odot g_t,a\right\rangle+D_{\psi}(a,a_t)+\rho D_{\psi}(a,a^{\mathrm{ref}})\right\}.$$

## Implementation notes

Integrate this into AdamW or SGD at the parameter-group learning-rate line, initially using one gain per layer or per optimizer parameter group rather than one gain per scalar weight. Maintain the usual optimizer state if using AdamW, but multiply its normalized update by the learned gain: let $u_t$ be the AdamW direction and set $\theta_{t+1}=\theta_t-a_t\odot u_t$. Store the previous direction and the current minibatch gradient. After applying the update using $u_t$, compute the next minibatch direction $u_{t+1}$ and form the delayed gain-gradient estimate $q_t=-\operatorname{stopgrad}(u_t)\odot u_{t+1}$. Average $q_t$ over each parameter group and optionally clip it to $[-q_{\max},q_{\max}]$. For the Euclidean mirror potential $\psi(a)=\frac12a^2$, implement the exact closed-form step $a_{t+1}=\operatorname{clip}((a_t-\eta q_t+\rho a^{\mathrm{ref}})/(1+\rho),a_{\min},a_{\max})$. Use $a_{\min}=0.1\alpha_0$, $a_{\max}=10\alpha_0$, and $a^{\mathrm{ref}}=\alpha_0$, where $\alpha_0$ is the baseline learning rate. The paper-derived quantities are the consecutive-direction product, bounded gain domain, mirror update, and Bregman pull; group averaging and clipping are practical approximations. First test on CIFAR-10 with a small six-layer CNN and a two-layer Transformer, comparing SGD or AdamW, unconstrained learned rates, and this optimizer at equal FLOPs. Record loss versus update count, validation accuracy, gain trajectories, clipping frequency, gradient-reversal frequency, and exploding-loss events. A successful result is faster loss descent or equal accuracy with fewer instabilities, particularly under deliberate learning-rate miscalibration or a distribution shift halfway through training.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a bounded predictive-gain optimizer with Euclidean mirror descent, groupwise delayed direction products, clipping, persistence, and a deterministic mini-training comparison. All three mechanism predictions were confirmed: gain response slope 0.08450000000000002 predicted versus 0.08450000000000005 observed, persistence ratio 0.6666666666666666 exactly matched, and strict quadratic sign reversal began on the first grid point above the predicted aλ=1 threshold while the stability interval test had zero mismatches. The toy baseline and adaptive optimizer had effectively identical final accuracy (0.93375), with final losses 0.1388764 and 0.1388563 respectively, so no meaningful optimization win was observed.

### Mechanism check

- Verdict: Built a bounded predictive-gain optimizer with Euclidean mirror descent, groupwise delayed direction products, clipping, persistence, and a deterministic mini-training comparison. All three mechanism predictions were confirmed: gain response slope 0.08450000000000002 predicted versus 0.08450000000000005 observed, persistence ratio 0.6666666666666666 exactly matched, and strict quadratic sign reversal began on the first grid point above the predicted aλ=1 threshold while the stability interval test had zero mismatches. The toy baseline and adaptive optimizer had effectively identical final accuracy (0.93375), with final losses 0.1388764 and 0.1388563 respectively, so no meaningful optimization win was observed.
- Confidence: 9/10
- Limitations: The experiment used a small deterministic NumPy MLP and synthetic 2D data, not CIFAR-10, a CNN, Transformer, AdamW, GPU execution, or equal-FLOP profiling. It did not test deliberate learning-rate miscalibration, distribution shift, scalar-per-parameter gains, or statistically replicated seeds.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/768/bench_report.json)
- [experiment.py](https://synthcore.org/code/768/experiment.py)
- [report.md](https://synthcore.org/code/768/report.md)
- [report_bench_2026-09-01T100719.md](https://synthcore.org/code/768/report_bench_2026-09-01T100719.md)
- [results.json](https://synthcore.org/code/768/results.json)
- [run_output.json](https://synthcore.org/code/768/run_output.json)
- [stage2_bench.py](https://synthcore.org/code/768/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/768)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.