✗ Failed on benchmark 2026

Decision-Oriented Optimum Preservation

Implementation & benchmark of arXiv:2608.23885 — A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization

Usefulness8/10

Difficulty6/10

Novelty7/10

Training Accuracy Stability Generalization Optimizer Loss World model Training dynamics Optimization Dynamical systems Control theory Statistics

Source paper: A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization arXiv:2608.23885 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Train a neural dynamical surrogate not only to reproduce measured trajectories, but also to reproduce the plant's economically optimal decision and objective value. Add a differentiable decision loss obtained by solving the surrogate's inner optimization problem, and reject models that fit observations while producing extra local optima or a shifted optimum.

Formulas

$$J_{\theta}(u)=c(x_{\theta}(u),u),\qquad u_{\theta}^{\star}=\arg\min_{u\in\mathcal U}J_{\theta}(u),$$

$$L(\theta)=L_{\mathrm{data}}(\theta)+\lambda_u\|u_{\theta}^{\star}-u^{\star}\|_{W_u}^{2}+\lambda_J\left|J_{\theta}(u_{\theta}^{\star})-J^{\star}\right|^{2},$$

$$\nabla_uJ_{\theta}(u_{\theta}^{\star})=0,\qquad \lambda_{\min}\!\left(\nabla_u^2J_{\theta}(u_{\theta}^{\star})\right)>0\ \text{for a strict local minimum}.$$

Mathematical statement

Let $f_{\theta}$ be a learned process model, $u\in\mathcal U$ the vector of operating decisions, $x_{\theta}(u)$ the predicted steady state or rollout, and $J_{\theta}(u)=c(x_{\theta}(u),u)$ the predicted economic objective to minimize. Let $u^\star$ and $J^\star$ be the trusted plant optimum from a benchmark or high-fidelity simulator. In addition to trajectory loss $L_{\mathrm{data}}(\theta)$, minimize $L_{\mathrm{dec}}(\theta)=\|u_{\theta}^\star-u^\star\|_{W_u}^2+\alpha|J_{\theta}(u_{\theta}^\star)-J^\star|^2$, where $u_{\theta}^\star=\arg\min_{u\in\mathcal U}J_{\theta}(u)$ and $W_u\succeq0$ weights decision coordinates. A local optimum must satisfy $\nabla_uJ_{\theta}(u)=0$ and $\nabla_u^2J_{\theta}(u)\succeq0$. The mechanism predicts that trajectory-equivalent models can have substantially different $u_{\theta}^\star$; decision loss removes this non-identifiability in directions relevant to control.

Implementation notes

Integrate this at the surrogate-model training level. Use an MLP or neural ODE to predict the state trajectory or steady state from decision vector $u$, then compute the economic objective $J_{\theta}(u)$ using the same differentiable cost and constraint code used by the real optimizer. For every minibatch or every $K$ training steps, solve the inner problem with projected L-BFGS, SQP, or differentiable unrolled gradient descent from several initial points. Pseudocode is: (1) predict $x_{\theta}(u)$ on measured data and compute $L_{\mathrm{data}}$; (2) initialize $M$ decision vectors; (3) repeat $u_m\leftarrow\Pi_{\mathcal U}[u_m-\rho\nabla_uJ_{\theta}(u_m)]$ for $T$ steps; (4) select the lowest feasible objective as $u_{\theta}^{\star}$; (5) backpropagate $L_{\mathrm{data}}+\lambda_u\|u_{\theta}^{\star}-u^\star\|^2+\lambda_J|J_{\theta}(u_{\theta}^{\star})-J^\star|^2$. The paper supplies the failure mechanism and the decision benchmark; the plant optimum, feasible set, and cost are supplied by the application, while gradients and Hessians are computed by autodiff and the number of optima is estimated by multistart. First experiment: fit a neural ODE on synthetic nonlinear process data where the true objective has one known optimum, compare MSE-only training with the added decision loss, and run 100 multistart optimizations. The quantitative prediction is that MSE-only models will sometimes have several local minima and a nonzero optimum displacement, while decision-aware training should reduce $\|u_{\theta}^{\star}-u^\star\|$ and the number of distinct converged optima; measure success as at least a 50% reduction in optimum displacement and no extra minima in the well-conditioned benchmark.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a differentiable surrogate with unrolled projected inner optimization and decision loss, plus fixed-seed math sweeps. The quadratic inner-loop stability boundary was confirmed at rho=2/H: predicted 0.5405, with contraction at 0.50 and divergence at 0.55. Objective perturbations produced extra minima as epsilon increased, and decision-aware training reduced optimum displacement from 0.40495 to 0.01505 (96.3%) with nearly unchanged data MSE. The mechanism manifested, but the stronger claim about removing extra minima was not established because the simple multistart gradient-flow count produced flat-region artifacts.

Agent confidence: 8/10

Baseline: MSE-only: optimum u=0.160, displacement=0.40495, objective=0.01587, data MSE=0.07961, grid minima=1; multistart endpoint count=19 but unreliable.
Idea: Decision-aware: optimum u=0.580, displacement=0.01505, objective=0.01005, data MSE=0.08007, grid minima=1; multistart endpoint count=14 but unreliable. Stability: predicted boundary rho=0.54054, observed transition between 0.50 stable and 0.55 divergent. Perturbation sweep: 1 minimum through epsilon=0.01, 2 at 0.02, 3 at 0.04.

Limitations: Only a one-dimensional synthetic steady-state surrogate was tested; no neural ODE, constraints beyond box projection, noisy trajectories, multiple decision dimensions, computational/FLOP comparison, or robust Hessian-filtered/local-root basin counting was performed. The simple multistart gradient-flow endpoint counts should not be interpreted as distinct local-minimum counts.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.23885) Sep 2, 2026

Significantly WORSE than the tuned baseline (+36148.3%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0021
Idea mean: 0.7633
Effect (Δ): 0.7612 (+36148.3%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±2233.5%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track was tested; no custom track, longer training budget, alternative decision-loss weights, or more extensive decision-probe reporting was run. The benchmark's standard task MSE is the primary result, and the decision objective used a nominal one-step pendulum cost rather than a full multi-step economic rollout.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Built a local decision-aware training loop for the structurally matched actuated-pendulum dynamics track using the same rnn_small architecture as the canonical MSE baseline. Across 8 paired seeds, the baseline achieved mean test MSE 0.002106 while the best decision-aware configuration achieved 0.763324; the paired delta was +0.761218 with permutation p=0.0081, a significant loss. The trained-model decision mechanism signature was not quantitatively confirmed, and the idea is therefore a clear negative result on this benchmark.

Baseline: dynamics/rnn_small, swept lr {0.001, 0.003, 0.006} at 12 epochs; best lr=0.006; 8-seed mean test MSE=0.0021058194, std=0.0003894366; per-seed=[0.0020043950, 0.0014378825, 0.0015748648, 0.0022957840, 0.0024020642, 0.0024898043, 0.0020851335, 0.0025566272]
Idea: Decision-aware unrolled projected multistart loss, same rnn_small/lr/epochs grid; best lr=0.001; 8-seed mean test MSE=0.7633235902, std=0.0525785229; per-seed=[0.7467728257, 0.7365002632, 0.8016765714, 0.8869475126, 0.7358804345, 0.7473666072, 0.7420515418, 0.7093929648]
Idea sweep: lr {0.001, 0.003, 0.006}, all 12 epochs; means={0.001:0.7633235902, 0.003:0.7637511864, 0.006:0.7635712028}
Paired comparison: delta_mean=+0.7612177707 (idea minus baseline; lower is better), idea_wins=0/8, permutation_p=0.0081, verdict='idea worse (significant)'

Benchmark result