✗ Failed on benchmark 2026

Key-Selective Delta Momentum

Implementation & benchmark of arXiv:2608.19491 — DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule

Usefulness8/10

Difficulty5/10

Novelty7/10

Optimization Speedup Stability Accuracy Optimizer Mlp Cnn Transformer Linear algebra Optimization Dynamical systems

Source paper: DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule arXiv:2608.19491 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the global EMA update for each linear-layer momentum matrix with a delta-rule update that learns the current output-side gradient value only along the current input-key direction. Frequently occurring directions are corrected repeatedly, while rarely visited directions are not unnecessarily overwritten or uniformly decayed. Use the resulting matrix as the ordinary momentum buffer in SGD, AdamW, or another optimizer.

Formulas

$$M_t^{\mathrm{EMA}}=\beta M_{t-1}^{\mathrm{EMA}}+(1-\beta)g_t,$$

$$g_t=\delta_t x_t^{\top}\in\mathbb{R}^{m\times n},\qquad y_t=W x_t,$$

$$M_t=M_{t-1}+\alpha\bigl(\delta_t-M_{t-1}x_t\bigr)x_t^{\top},$$

$$M_t=M_{t-1}+\alpha\frac{\bigl(\delta_t-M_{t-1}x_t\bigr)x_t^{\top}}{\|x_t\|_2^2+\varepsilon}.$$

Mathematical statement

The paper gives the EMA rule $M_t^{\mathrm{EMA}}=\beta M_{t-1}^{\mathrm{EMA}}+(1-\beta)g_t$, where $M_t$ is the momentum matrix, $g_t$ is the current gradient, and $\beta\in[0,1)$ is the global decay. For a linear map $y=Wx$, with $W\in\mathbb{R}^{m\times n}$, input key $x\in\mathbb{R}^n$, and output-side error or value $\delta=\partial L/\partial y\in\mathbb{R}^m$, the per-sample gradient is the rank-one association $g_t=\delta_t x_t^\top$. The classical delta rule updates an associative-memory matrix by moving its current prediction $M_{t-1}x_t$ toward the target value $\delta_t$: $M_t=M_{t-1}+\alpha(\delta_t-M_{t-1}x_t)x_t^\top$, where $\alpha>0$ is the delta learning rate. A normalized version is $M_t=M_{t-1}+\alpha(\delta_t-M_{t-1}x_t)x_t^\top/(\|x_t\|_2^2+\varepsilon)$, with $\varepsilon>0$ preventing division by zero. The residual is nonzero when the stored momentum predicts the current value poorly, so the update is anisotropic and key-selective.

Implementation notes

Integrate this at the momentum-buffer update for every affine weight matrix, before the optimizer parameter update. For a minibatch linear layer with input activations $X\in\mathbb{R}^{B\times n}$ and output backpropagated errors $D\in\mathbb{R}^{B\times m}$, compute the ordinary gradient $G=D^\top X/B$ for the parameter update, but update the momentum buffer from per-example key-value pairs rather than from $G$ alone. A direct pseudocode implementation is: M = state.momentum; for b in 1..B: x = X[b]; delta = D[b]; pred = M @ x; residual = delta - pred; M += alpha * outer(residual, x) / (dot(x, x) + eps); state.momentum = M. A vectorized implementation computes P = X @ M.T, R = D - P, q = (X * X).sum(dim=1) + eps, and M += alpha * (R.T @ (X / q[:,None])). For AdamW, maintain the usual elementwise second moment V = beta2 * V + (1-beta2) * G**2, replace only the first-moment update by the delta buffer, and apply theta -= lr * M_hat / sqrt(V_hat + eps) followed by decoupled weight decay. Initialize alpha = 1 - beta1, then sweep alpha over 0.01, 0.03, 0.1, 0.3, and 1.0. The paper supplies the key-value factorization and delta-rule mechanism; estimate the practical coefficient, normalization, and whether sequential or parallel minibatch processing works better. First test a two-layer MLP and ResNet-18 on CIFAR-10 against SGD with momentum and AdamW under equal batch size and learning-rate tuning budgets. Record validation accuracy, loss versus optimizer steps, wall-clock throughput, and gradient-tracking error ||D - X M.T||_F / ||D||_F. Success means faster loss reduction at equal FLOPs, lower tracking error after input-distribution shifts, or matching baseline accuracy in fewer steps despite the extra matrix-vector and residual computations.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a normalized sequential Key-Selective Delta Momentum buffer and a reproducible toy regression experiment. The mechanism predictions were confirmed: residual contraction factors matched the exact formula (including the stability boundary alpha≈2 for ||x||²=5), orthogonal-key interference was 0, and old-direction retention was 1.0 for delta versus the EMA prediction/observation 0.9^10=0.348678. In the mini-experiment, best delta alpha=0.003 reached loss 0.002243 at step 150 versus momentum 0.005413 and final weight MSE 5.39e-6 versus 6.34e-6, but this is not evidence of a general speed/FLOP win because the delta method performs 64 sequential matrix updates per batch.

Agent confidence: 8/10

Baseline: SGD-style EMA momentum: loss@150=0.005413, loss@300=0.002241, final weight MSE=6.34e-6
Idea: Best normalized delta alpha=0.003: loss@150=0.002243, loss@300=0.002236, final weight MSE=5.39e-6; alpha sweep [0.001,0.003,0.01,0.03,0.1,0.3] showed degradation at extremes

Limitations: Only a 2D synthetic linear regression was tested; no MLP, CNN/ResNet, CIFAR-10, AdamW, wall-clock/FLOP accounting, vectorized minibatch implementation, or systematic learning-rate retuning was evaluated. The toy uses sequential updates and a deliberately shifted input distribution, so the observed accuracy/loss signal may not transfer to neural networks.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.19491) Sep 2, 2026

Significantly WORSE than the tuned baseline (+230.5%, p=0.0081, wins 0 / 0 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.6434
Idea mean: 0
Effect (Δ): 17.6159 (+230.5%; negative = idea better)
Wins: 0 / 0 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±24.1%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 0 configs

Limitations:

Only the structurally matched tabular track and mlp_tiny architecture were tested. No vision, sequence, dynamics, AdamW, throughput, or FLOP comparison was run. The mechanism-signature probe failed numerically with NaN, so it provides no confirmation.

How to run: python3 stage2_delta_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented Key-Selective Delta Momentum as a custom optimizer loop for the structurally matched tabular Friedman#1 MLP track, using eight paired seeds and a shared learning-rate search space. The best idea configuration (lr=0.01, alpha=0.001) achieved mean test MSE 25.2594 versus 7.6434 for tuned SGD momentum; paired permutation p=0.0081 and positive delta_mean indicate the idea was significantly worse. The trained-model mechanism signature returned NaN and was not confirmed.

Baseline: SGD global EMA momentum, best lr=0.01: mean test MSE 7.6434266567 over 8 seeds
Idea: Normalized key-selective delta momentum, best lr=0.01 alpha=0.001: mean test MSE 25.2593717575 over 8 seeds
Paired delta: idea - baseline = +17.6159451008 MSE; permutation p=0.0081; verdict: idea worse (significant)

Benchmark result