✗ Failed on benchmark 2026

Projection-Regularized Gradient Updates

Implementation & benchmark of arXiv:2607.28123 — Projection-Regularized Indirect Data-Driven Predictive Control

Usefulness7/10

Difficulty6/10

Novelty6/10

Optimizer Stability Generalization Speedup Optimizer Training dynamics Fine tuning Optimization Linear algebra Statistics Dynamical systems

Source paper: Projection-Regularized Indirect Data-Driven Predictive Control arXiv:2607.28123 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace unconstrained neural-network updates by updates projected toward directions supported by a recent, regularized gradient or feature subspace. This transfers PRPC's errors-in-variables correction: directions that are weakly identified by noisy or rank-deficient minibatches receive stronger shrinkage, preventing large updates caused by accidental correlations. The method is especially suitable for recurrent, world-model, and small-data fine-tuning problems where minibatch covariance is ill-conditioned.

Formulas

$$S_t=\beta S_{t-1}+(1-\beta)\frac{G_tG_t^\top}{m}+\lambda I_p,\qquad G_t=[g_{t-m+1},\ldots,g_t].$$

$$P_t=S_t^{-1/2}G_t\left(G_t^\top S_t^{-1}G_t+\rho I_m\right)^{-1}G_t^\top S_t^{-1/2}.$$

$$\theta_{t+1}=\theta_t-\eta_tS_t^{-1/2}\left[(1-\alpha)P_t+\alpha I_p\right]S_t^{-1/2}g_t.$$

$$0<\eta_t q_i h_i<2\quad\text{for every local Hessian mode }i.$$

Mathematical statement

Let $g_t=\nabla_\theta \ell_t(\theta_t)\in\mathbb{R}^p$ be the minibatch gradient, let $G_t=[g_{t-m+1},\ldots,g_t]\in\mathbb{R}^{p\times m}$ collect the last $m$ gradients, and define the regularized covariance $S_t=\beta S_{t-1}+(1-\beta)G_tG_t^\top/m+\lambda I_p$, where $\beta\in[0,1)$ and $\lambda>0$. The data-supported projection is $P_t=S_t^{-1/2}G_t(G_t^\top S_t^{-1}G_t+\rho I_m)^{-1}G_t^\top S_t^{-1/2}$, with projection strength $\rho>0$. Use the preconditioned/projected direction $d_t=S_t^{-1/2}[(1-\alpha)P_t+\alpha I_p]S_t^{-1/2}g_t$, where $\alpha\in[0,1]$. The update is $\theta_{t+1}=\theta_t-\eta_t d_t$. Eigen-directions with small empirical support are controlled by $\lambda$ and cannot produce arbitrarily large steps. For a locally quadratic loss with Hessian eigenvalue $h_i$, stability requires approximately $0<\eta_t q_i h_i<2$ in every effective direction, where $q_i$ is the corresponding eigenvalue of the update metric.

Implementation notes

Integrate this as an optimizer wrapper around AdamW or SGD, initially without momentum or weight decay so that the effect of the projection is isolated. At step $t$, compute the ordinary minibatch gradient $g_t$, maintain a rolling matrix of the last $m$ flattened gradients, and update the exponentially averaged covariance $S_t$. For a first implementation, avoid explicit $p\times p$ matrices: use a diagonal approximation for $S_t$ in large models, or apply the Woodbury identity to compute products with the low-rank matrix $G_tG_t^\top$. Compute the projected direction by solving the $m\times m$ system $G_t^\top S_t^{-1}G_t+\rho I_m$; never form a pseudoinverse. Apply gradient clipping only after projection, then update parameters with learning rate $\eta_t$. The quantities $G_t$, $S_t$, and $P_t$ are empirical estimates; the projection formula and ridge regularization are the transferred paper mechanism. Start with a two-layer MLP on CIFAR-10 or a small transformer on a noisy synthetic sequence task, comparing SGD or AdamW against the projected optimizer at matched step sizes. Measure the largest stable learning rate by sweeping $\eta$. The quantitative prediction is that the divergence threshold shifts according to $\eta_c\approx2/(q_{\max}h_{\max})$ rather than the unprojected $2/h_{\max}$, and that validation-loss variance decreases most strongly when the gradient covariance has small effective rank. Estimate local $h_i$ with Hessian-vector products and test whether the measured threshold agrees within 20%.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an exact small-model ProjectionRegularizedSGD optimizer and a deterministic quadratic verification harness. The measured stability thresholds matched the analytical thresholds to about 1e-9; increasing lambda and rho monotonically increased the threshold, while alpha=0 gave exactly zero unsupported-direction metric and increasing alpha increased it. In the noisy quadratic comparison at eta=0.055, the projected method reached mean final loss 0.0232 versus 1.511 for the nonprojected regularized baseline, but this is secondary because that baseline is not plain SGD.

Agent confidence: 9/10

Alpha support: Unsupported q: alpha=0: 0.0; .02: 0.2; .1: 1.0; .3: 3.0; 1.0: 10.0
Baseline: Noisy quadratic, nonprojected regularized metric (alpha=1): mean final loss 1.5106 +/- 1.1749 over 8 seeds
Idea: Noisy quadratic, projection-regularized metric (lambda=0.1, rho=0.15, alpha=0.08): mean final loss 0.02316 +/- 0.00415 over 8 seeds
Rho thresholds: Predicted/observed eta_c: rho=.01 0.17920330/0.17920330; .03 0.18010930/0.18010930; .1 0.18327402/0.18327402; .3 0.19226259/0.19226259; 1.0 0.22311154/0.22311154
Ridge thresholds: Predicted/observed eta_c: lambda=.01 0.16944022/0.16944022; .03 0.17300721/0.17300721; .1 0.18552857/0.18552857; .3 0.22161991/0.22161991; 1.0 0.35161534/0.35161534

Limitations: Only a 2D deterministic/noisy quadratic and a small PyTorch smoke test were evaluated; no CIFAR, recurrent, transformer, or realistic small-data fine-tuning experiment was run. The optimizer uses explicit p-by-p matrices and is limited to small models; no wall-clock, FLOP, Hessian-vector-product, clipping, AdamW, or plain-SGD comparison was tested.

How to run: python3 projection_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.28123) Sep 2, 2026

Significantly WORSE than the tuned baseline (+41.9%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.9055
Idea mean: 11.2195
Effect (Δ): 3.314 (+41.9%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±9.6%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the mandated tabular track was tested; vision, sequence, and dynamics were not applicable to this optimizer idea. The implementation uses a diagonal approximation rather than the exact dense/low-rank matrix formulation, and no wall-clock/FLOP comparison, Hessian-vector-product stability sweep, AdamW, clipping, or alternative projection hyperparameter sweep beyond the fixed idea settings was performed.

How to run: python3 bench_projection.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Built a diagonal scalable Projection-Regularized Gradient optimizer and evaluated it against matched Adam on the Friedman#1 tabular track using the same mlp_tiny architecture, data, epochs, and learning-rate union. Across 8 paired seeds, tuned Adam achieved test MSE 7.9055 while the best projection setting achieved 11.2195, with delta_mean +3.3140 and permutation p=0.0081; therefore the idea was significantly worse. The trained-model signature showed nonzero empirical support-dependent metric modulation, but this mechanism signal did not translate into a task-metric win.

Baseline: Tuned Adam, mlp_tiny, lr=0.01: test MSE 7.905456 +/- 0.733317 over 8 seeds
Idea: Diagonal projection optimizer, mlp_tiny, lr=0.01, m=8, beta=0.9, lambda=0.1, rho=0.15, alpha=0.08: test MSE 11.219475 +/- 0.933073 over 8 seeds
Idea wins: 0/8
Mechanism signature: Observed support_fraction=0.514455, q_mean=0.473345, grad_norm=44.269749; support-dependent shrinkage was observed, but the original report's confirmed flag is not treated as evidence of task improvement.
Paired delta: idea - baseline = +3.314019 MSE
Permutation p: 0.0081

Benchmark result