✓✓ Beats tuned baseline 2026

State-Dependent Metric Projected Optimizer

Implementation & benchmark of arXiv:2607.14519 — State-Dependent Metric Projection Neural Network for Variational Inequalities

Usefulness7/10

Difficulty5/10

Novelty6/10

Optimization Speedup Stability Accuracy Optimizer Training Optimization Convex analysis Geometry Dynamical systems

Source paper: State-Dependent Metric Projection Neural Network for Variational Inequalities arXiv:2607.14519 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the usual projected gradient step with a relaxed projection in a positive-definite metric that changes with the current parameter state. The metric acts as a continuous preconditioner before projection, so updates can be large along poorly conditioned directions while remaining feasible.

Formulas

$$\mathrm{VI}(F,\mathcal{S}):\qquad \text{find }\bar{x}\in\mathcal{S}\text{ such that }\langle F(\bar{x}),x-\bar{x}\rangle\geq 0,\quad\forall x\in\mathcal{S}.$$

$$P_{\mathcal{S}}^{M(x)^{-1}}(z)=\underset{y\in\mathcal{S}}{\arg\min}\;\frac{1}{2}(y-z)^\top M(x)^{-1}(y-z).$$

$$\frac{dx}{dt}=\lambda\left(P_{\mathcal{S}}^{M(x)^{-1}}\left(x-\alpha M(x)F(x)\right)-x\right).$$

$$\theta_{k+1}=\theta_k+\eta\left[P_{\mathcal{S}}^{M(\theta_k)^{-1}}\left(\theta_k-\alpha M(\theta_k)\,g_k\right)-\theta_k\right],\qquad g_k=\nabla_\theta\mathcal{L}(\theta_k).$$

Mathematical statement

The paper studies the variational inequality VI(F,S): find x_bar in the closed convex feasible set S such that <F(x_bar), x-x_bar> >= 0 for every x in S. For a symmetric positive-definite matrix M(x), define the weighted projection P_S^{M(x)^(-1)}(z) = argmin_{y in S} (1/2)(y-z)^T M(x)^(-1)(y-z). The central state-dependent metric projection flow is dx/dt = lambda[P_S^{M(x)^(-1)}(x-alpha M(x)F(x))-x], where F is the operator or gradient, alpha > 0 is the step scale, and lambda > 0 is the relaxation rate. At equilibrium x = P_S^{M(x)^(-1)}(x-alpha M(x)F(x)); the projection optimality condition implies <F(x), y-x> >= 0 for every y in S, because M(x)^(-1)(x-alpha M(x)F(x)-x) = -alpha F(x). Thus any equilibrium solves the original VI independently of the particular positive-definite metric. For neural optimization, use F(theta) = gradient of the training loss and a diagonal metric M(theta) = diag(m_i(theta)) with m_i(theta) > 0, for example m_i(theta) = exp(clip(a_i(theta), log(m_min), log(m_max))).

Implementation notes

(1) Integration point: modify the optimizer update immediately after computing the minibatch gradient g_k, and before any ordinary parameter projection or clipping. Start with box constraints S = [l,u]^d so the weighted projection is coordinatewise exact. Maintain a positive diagonal metric M(theta_k) = diag(m_1,...,m_d), with m_i = exp(clamp(a_i, log(m_min), log(m_max))). A simple first choice is a_i = beta log(v_i + epsilon), where v_i is Adam's bias-corrected second moment; alternatively use a small learned function of log(|theta_i|+epsilon) and log(v_i+epsilon). Compute the preconditioned proposal z_i = theta_i - alpha*m_i*g_i. For a box, compute p_i = min(u_i, max(l_i, z_i)); then apply relaxation theta_i <- theta_i + eta*(p_i-theta_i). (2) Pseudocode: initialize theta, moment statistics v, alpha, eta; for each minibatch compute g; update v if using adaptive metrics; set m=exp(clamp(beta*log(v+epsilon), log(m_min), log(m_max))); z=theta-alpha*m*g; p=clamp(z,l,u); theta=theta+eta*(p-theta). Do not use Adam's usual division by sqrt(v) in the same branch, otherwise the effect of M is confounded; compare against projected SGD and projected Adam with matched effective step sizes. (3) The mathematical quantities are the SPD metric, weighted projection, and relaxed projected flow. Estimate v empirically from minibatch gradients, and estimate conditioning improvement from gradient covariance or Hessian-vector products; enforce m_min > 0 and m_max < infinity for stability. For general polytope constraints, replace clamp with a quadratic-program solver for the weighted projection. (4) First experiment: train a small MLP on a synthetic strongly convex quadratic with diagonal Hessian eigenvalues spanning 10^4, impose box bounds, and compare projected SGD, projected Adam, and this method at equal gradient evaluations. Measure loss versus steps, constraint violations, final objective, and sensitivity to alpha. A successful result is faster loss decrease or lower loss at equal FLOPs, with zero constraint violations and no increase in update oscillation.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a NumPy MVP for state-dependent diagonal metric projected optimization with exact box projection, numerical KKT verification, and ill-conditioned quadratic experiments. The mechanism predictions were confirmed: contraction ratios matched |1-eta*alpha*m*h| to numerical precision, the predicted stability boundary 2/(eta*h)=0.05405 was observed near 0.0561, and fractional decrease scaled linearly with metric m. The optimizer comparison was not a general win: positive beta caused saturation and loss 20000, while inverse-variance beta=-0.5 at alpha=0.01 reached 0.09 versus SGD 0.47 but remained worse than projected Adam 1.56e-5.

Agent confidence: 8/10

Baseline: Projected SGD final quadratic loss 0.473714 after 180 steps; projected Adam 1.562e-5; all observed box violations were 0.
Idea: Default state metric beta=+0.5, alpha=0.03 final loss 20000.001; sensitivity found beta=-0.5, alpha=0.01 reached 0.08999 with zero violations, while larger steps often diverged into the box boundary.

Limitations: Only a 2D deterministic quadratic and exact box constraints were tested; no neural-network minibatch training, general polytope QP projection, wall-clock or FLOP measurement, stochastic gradients, or broad hyperparameter tuning was performed. The observed stability boundary uses a finite-run divergence threshold and is therefore approximate.

How to run: python3 metric_projected_optimizer.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.14519) Sep 2, 2026

Beats the tuned baseline by 11.5% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 17.6662
Idea mean: 15.6386
Effect (Δ): -2.0276 (−11.5%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±5.6%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched registered tabular track was tested. General polytope projection, wall-clock/FLOP measurements, alternative architectures, and broader metric parameterizations were not tested.

How to run: python3 stage2_metric_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Worked ✓

Agent confidence: 8/10

The registered tabular Friedman regression benchmark used the same mlp_tiny architecture and matched learning-rate union for tuned Adam baseline versus the state-dependent metric projected optimizer. The idea improved mean test MSE from 17.666198 to 15.638604 across all 8 paired seeds, with delta_mean=-2.027594 and permutation p_value=0.0081, giving the bench verdict idea better (significant).

Baseline: Tuned Adam mean test MSE 17.666198.
Idea: State-dependent metric projection mean test MSE 15.638604; 8/8 paired wins; zero constraint violations; mechanism correlation 0.807046 and confirmed=true.

Benchmark result