✗ Failed on benchmark 2026

Bounded predictive-gain optimizer

Implementation & benchmark of arXiv:2608.09218 — Online Learning of Scale Parameters in Score-Driven Filters

Usefulness8/10

Difficulty5/10

Novelty6/10

Optimization Speedup Stability Generalization Optimizer Training dynamics Fine tuning Optimization Convex analysis Information theory Dynamical systems

Source paper: Online Learning of Scale Parameters in Score-Driven Filters arXiv:2608.09218 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a fixed learning rate for each layer or parameter block with a bounded gain selected by the one-step-ahead predictive loss. The sign of the product between the current gradient and the next gradient estimates whether the previous update moved downhill: aligned gradients increase the gain, while sign reversals decrease it. A mirror-descent update on a bounded interval prevents the runaway step sizes that can occur with exponential or unconstrained learning-rate parameterizations.

Formulas

$$P_{\lambda}(\mathrm{d}y)=p(y\mid\lambda)\nu(\mathrm{d}y),\qquad \lambda\in\Lambda\subset\mathbb{R}^{m}.$$

$$\theta_{t+1}=\theta_t-a_t\odot g_t,\qquad g_t=\nabla_{\theta}f_t(\theta_t),$$

$$\widehat{\nabla}_{a_t}\ell_t=-g_{t+1}\odot g_t,$$

$$a_{t+1}=\arg\min_{a\in[a_{\min},a_{\max}]^d}\left\{\eta\left\langle-g_{t+1}\odot g_t,a\right\rangle+D_{\psi}(a,a_t)+\rho D_{\psi}(a,a^{\mathrm{ref}})\right\}.$$

Mathematical statement

The paper models a conditional predictive family as $P_{\lambda}(\mathrm{d}y)=p(y\mid\lambda)\nu(\mathrm{d}y)$, where $\lambda\in\Lambda\subset\mathbb{R}^{m}$ is the state and $\nu$ is a dominating measure. A score-driven state update has the generic form $\lambda_{t+1}=\omega(\lambda_t)+G_t s_t$, where $s_t=\nabla_{\lambda}\log p(y_t\mid\lambda_t)$ is the score and $G_t$ is a gain. The paper's central observation is that the negative product of consecutive scores is the stochastic gradient of a one-step predictive loss with respect to the gain. For neural-network loss $f_t(\theta)$, let $g_t=\nabla_{\theta}f_t(\theta_t)$ and update $\theta_{t+1}=\theta_t-a_t\odot g_t$. The gain derivative is estimated by $\widehat{\nabla}_{a_t}\ell_t=-g_{t+1}\odot g_t$, because increasing $a_{t,i}$ changes the next parameter by $-g_{t,i}$ and the observed next loss gradient is $g_{t+1,i}$. Thus aligned consecutive gradients increase the gain and gradient reversals decrease it. Let $a_i\in[a_{\min},a_{\max}]$, let $\psi$ be differentiable and strictly convex, and define $D_{\psi}(x,y)=\psi(x)-\psi(y)-\langle\nabla\psi(y),x-y\rangle$. With reference gain $a^{\mathrm{ref}}$ and persistence coefficient $\rho\ge0$, the mirror update is $a_{t+1}=\arg\min_{a\in[a_{\min},a_{\max}]^d}\{\eta\langle\widehat{\nabla}_{a_t}\ell_t,a\rangle+D_{\psi}(a,a_t)+\rho D_{\psi}(a,a^{\mathrm{ref}})\}$. Here $d$ is the number of gain groups and $\eta$ is the gain learning rate.

Implementation notes

Integrate this into AdamW or SGD at the parameter-group learning-rate line, initially using one gain per layer or per optimizer parameter group rather than one gain per scalar weight. Maintain the usual optimizer state if using AdamW, but multiply its normalized update by the learned gain: let $u_t$ be the AdamW direction and set $\theta_{t+1}=\theta_t-a_t\odot u_t$. Store the previous direction and the current minibatch gradient. After applying the update using $u_t$, compute the next minibatch direction $u_{t+1}$ and form the delayed gain-gradient estimate $q_t=-\operatorname{stopgrad}(u_t)\odot u_{t+1}$. Average $q_t$ over each parameter group and optionally clip it to $[-q_{\max},q_{\max}]$. For the Euclidean mirror potential $\psi(a)=\frac12a^2$, implement the exact closed-form step $a_{t+1}=\operatorname{clip}((a_t-\eta q_t+\rho a^{\mathrm{ref}})/(1+\rho),a_{\min},a_{\max})$. Use $a_{\min}=0.1\alpha_0$, $a_{\max}=10\alpha_0$, and $a^{\mathrm{ref}}=\alpha_0$, where $\alpha_0$ is the baseline learning rate. The paper-derived quantities are the consecutive-direction product, bounded gain domain, mirror update, and Bregman pull; group averaging and clipping are practical approximations. First test on CIFAR-10 with a small six-layer CNN and a two-layer Transformer, comparing SGD or AdamW, unconstrained learned rates, and this optimizer at equal FLOPs. Record loss versus update count, validation accuracy, gain trajectories, clipping frequency, gradient-reversal frequency, and exploding-loss events. A successful result is faster loss descent or equal accuracy with fewer instabilities, particularly under deliberate learning-rate miscalibration or a distribution shift halfway through training.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a bounded predictive-gain optimizer with Euclidean mirror descent, groupwise delayed direction products, clipping, persistence, and a deterministic mini-training comparison. All three mechanism predictions were confirmed: gain response slope 0.08450000000000002 predicted versus 0.08450000000000005 observed, persistence ratio 0.6666666666666666 exactly matched, and strict quadratic sign reversal began on the first grid point above the predicted aλ=1 threshold while the stability interval test had zero mismatches. The toy baseline and adaptive optimizer had effectively identical final accuracy (0.93375), with final losses 0.1388764 and 0.1388563 respectively, so no meaningful optimization win was observed.

Agent confidence: 9/10

Baseline: final_loss=0.1388764, best_loss=0.0571861, final_accuracy=0.93375, loss_at_40=0.1623599
Idea: final_loss=0.1388563, best_loss=0.0571893, final_accuracy=0.93375, loss_at_40=0.1624318, final_gains=[0.7997841,0.7996852], reversal_events=185, clip_events=0

Limitations: The experiment used a small deterministic NumPy MLP and synthetic 2D data, not CIFAR-10, a CNN, Transformer, AdamW, GPU execution, or equal-FLOP profiling. It did not test deliberate learning-rate miscalibration, distribution shift, scalar-per-parameter gains, or statistically replicated seeds.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.09218) Sep 1, 2026

Significantly WORSE than the tuned baseline (+212.5%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.6676
Idea mean: 23.9583
Effect (Δ): 16.2907 (+212.5%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±49.2%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the structurally matched tabular track was tested. Vision, sequence, dynamics, AdamW, deliberate learning-rate miscalibration, distribution shift, longer training, and equal-FLOP profiling were not tested. The implementation used layerwise gains for SGD rather than AdamW normalized directions.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

The bounded predictive-gain optimizer was implemented as a groupwise Euclidean mirror-descent modification of SGD and tested on the registered tabular Friedman#1 regression track with the fixed mlp_tiny architecture. The analytic update check passed to numerical precision, but the idea was significantly worse on test MSE: 23.9583 versus 7.6676 for baseline, paired delta +16.2907, permutation p=0.0081. The trained-model mechanism signature was not confirmed, so there is no observed transfer win.

Baseline: mean test MSE=7.6675900817, std=1.7205125356, best_cfg={lr:0.006,wd:0.0}
Idea: mean test MSE=23.9582793713, std=3.6945581467, selected_cfg={lr:0.003,gain_eta:0.08,rho:0.15}
Paired delta: 16.2906892896 (idea minus baseline; lower is better)
Permutation p: 0.0081

Benchmark result