# State-Dependent Metric Projected Optimizer

- ID: 918
- Canonical URL: https://synthcore.org/idea/918/state-dependent-metric-projected-optimizer
- API JSON: https://synthcore.org/api/idea/918.json
- API Markdown: https://synthcore.org/api/idea/918.md
- Verification status: beats_baseline
- Source: [arXiv:2607.14519](https://arxiv.org/abs/2607.14519)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, training
- Math tags: optimization, convex-analysis, geometry, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace the usual projected gradient step with a relaxed projection in a positive-definite metric that changes with the current parameter state. The metric acts as a continuous preconditioner before projection, so updates can be large along poorly conditioned directions while remaining feasible.

## Mathematical statement

The paper studies the variational inequality VI(F,S): find x_bar in the closed convex feasible set S such that <F(x_bar), x-x_bar> >= 0 for every x in S. For a symmetric positive-definite matrix M(x), define the weighted projection P_S^{M(x)^(-1)}(z) = argmin_{y in S} (1/2)(y-z)^T M(x)^(-1)(y-z). The central state-dependent metric projection flow is dx/dt = lambda[P_S^{M(x)^(-1)}(x-alpha M(x)F(x))-x], where F is the operator or gradient, alpha > 0 is the step scale, and lambda > 0 is the relaxation rate. At equilibrium x = P_S^{M(x)^(-1)}(x-alpha M(x)F(x)); the projection optimality condition implies <F(x), y-x> >= 0 for every y in S, because M(x)^(-1)(x-alpha M(x)F(x)-x) = -alpha F(x). Thus any equilibrium solves the original VI independently of the particular positive-definite metric. For neural optimization, use F(theta) = gradient of the training loss and a diagonal metric M(theta) = diag(m_i(theta)) with m_i(theta) > 0, for example m_i(theta) = exp(clip(a_i(theta), log(m_min), log(m_max))).

## Key formulas

- $$\mathrm{VI}(F,\mathcal{S}):\qquad \text{find }\bar{x}\in\mathcal{S}\text{ such that }\langle F(\bar{x}),x-\bar{x}\rangle\geq 0,\quad\forall x\in\mathcal{S}.$$
- $$P_{\mathcal{S}}^{M(x)^{-1}}(z)=\underset{y\in\mathcal{S}}{\arg\min}\;\frac{1}{2}(y-z)^\top M(x)^{-1}(y-z).$$
- $$\frac{dx}{dt}=\lambda\left(P_{\mathcal{S}}^{M(x)^{-1}}\left(x-\alpha M(x)F(x)\right)-x\right).$$
- $$\theta_{k+1}=\theta_k+\eta\left[P_{\mathcal{S}}^{M(\theta_k)^{-1}}\left(\theta_k-\alpha M(\theta_k)\,g_k\right)-\theta_k\right],\qquad g_k=\nabla_\theta\mathcal{L}(\theta_k).$$

## Implementation notes

(1) Integration point: modify the optimizer update immediately after computing the minibatch gradient g_k, and before any ordinary parameter projection or clipping. Start with box constraints S = [l,u]^d so the weighted projection is coordinatewise exact. Maintain a positive diagonal metric M(theta_k) = diag(m_1,...,m_d), with m_i = exp(clamp(a_i, log(m_min), log(m_max))). A simple first choice is a_i = beta log(v_i + epsilon), where v_i is Adam's bias-corrected second moment; alternatively use a small learned function of log(|theta_i|+epsilon) and log(v_i+epsilon). Compute the preconditioned proposal z_i = theta_i - alpha*m_i*g_i. For a box, compute p_i = min(u_i, max(l_i, z_i)); then apply relaxation theta_i <- theta_i + eta*(p_i-theta_i). (2) Pseudocode: initialize theta, moment statistics v, alpha, eta; for each minibatch compute g; update v if using adaptive metrics; set m=exp(clamp(beta*log(v+epsilon), log(m_min), log(m_max))); z=theta-alpha*m*g; p=clamp(z,l,u); theta=theta+eta*(p-theta). Do not use Adam's usual division by sqrt(v) in the same branch, otherwise the effect of M is confounded; compare against projected SGD and projected Adam with matched effective step sizes. (3) The mathematical quantities are the SPD metric, weighted projection, and relaxed projected flow. Estimate v empirically from minibatch gradients, and estimate conditioning improvement from gradient covariance or Hessian-vector products; enforce m_min > 0 and m_max < infinity for stability. For general polytope constraints, replace clamp with a quadratic-program solver for the weighted projection. (4) First experiment: train a small MLP on a synthetic strongly convex quadratic with diagonal Hessian eigenvalues spanning 10^4, impose box bounds, and compare projected SGD, projected Adam, and this method at equal gradient evaluations. Measure loss versus steps, constraint violations, final objective, and sensitivity to alpha. A successful result is faster loss decrease or lower loss at equal FLOPs, with zero constraint violations and no increase in update oscillation.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built a NumPy MVP for state-dependent diagonal metric projected optimization with exact box projection, numerical KKT verification, and ill-conditioned quadratic experiments. The mechanism predictions were confirmed: contraction ratios matched |1-eta*alpha*m*h| to numerical precision, the predicted stability boundary 2/(eta*h)=0.05405 was observed near 0.0561, and fractional decrease scaled linearly with metric m. The optimizer comparison was not a general win: positive beta caused saturation and loss 20000, while inverse-variance beta=-0.5 at alpha=0.01 reached 0.09 versus SGD 0.47 but remained worse than projected Adam 1.56e-5.

### Mechanism check

- Verdict: Built a NumPy MVP for state-dependent diagonal metric projected optimization with exact box projection, numerical KKT verification, and ill-conditioned quadratic experiments. The mechanism predictions were confirmed: contraction ratios matched |1-eta*alpha*m*h| to numerical precision, the predicted stability boundary 2/(eta*h)=0.05405 was observed near 0.0561, and fractional decrease scaled linearly with metric m. The optimizer comparison was not a general win: positive beta caused saturation and loss 20000, while inverse-variance beta=-0.5 at alpha=0.01 reached 0.09 versus SGD 0.47 but remained worse than projected Adam 1.56e-5.
- Confidence: 8/10
- Limitations: Only a 2D deterministic quadratic and exact box constraints were tested; no neural-network minibatch training, general polytope QP projection, wall-clock or FLOP measurement, stochastic gradients, or broad hyperparameter tuning was performed. The observed stability boundary uses a finite-run divergence threshold and is therefore approximate.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/381/bench_report.json)
- [metric_projected_optimizer.py](https://synthcore.org/code/381/metric_projected_optimizer.py)
- [report.md](https://synthcore.org/code/381/report.md)
- [report_bench_2026-08-31T095654.md](https://synthcore.org/code/381/report_bench_2026-08-31T095654.md)
- [results.json](https://synthcore.org/code/381/results.json)
- [sensitivity.json](https://synthcore.org/code/381/sensitivity.json)
- [stage2_metric_bench.py](https://synthcore.org/code/381/stage2_metric_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/381)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.