{
 "artifacts": [
  {
   "name": "bench_phantom.py",
   "url": "https://synthcore.org/code/1045/bench_phantom.py"
  },
  {
   "name": "bench_report.json",
   "url": "https://synthcore.org/code/1045/bench_report.json"
  },
  {
   "name": "phantom_audit.py",
   "url": "https://synthcore.org/code/1045/phantom_audit.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/1045/report.md"
  },
  {
   "name": "report_bench_2026-09-01T213431.md",
   "url": "https://synthcore.org/code/1045/report_bench_2026-09-01T213431.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/1045/results.json"
  }
 ],
 "category": "dynamics",
 "description": "Treat the optimized surrogate and the training trajectory as objects that require a decision-level audit. Use multistart optimization to count phantom optima, and periodically evaluate whether stochastic training has changed the surrogate optimum even when validation prediction error remains nearly constant; stop, roll back, or average checkpoints when decision drift exceeds a threshold.",
 "download_zip": "https://synthcore.org/download/1045",
 "formulas_latex": [
  "$$\\Phi(\\theta)=\\left\\{u\\in\\mathcal U:\\nabla_uJ_{\\theta}(u)=0,\\ \\nabla_u^2J_{\\theta}(u)\\succeq0\\right\\},\\qquad N(\\theta)=|\\Phi(\\theta)|,$$",
  "$$e_t=\\left\\|u_t^{\\star}-u_{\\mathrm{ref}}^{\\star}\\right\\|_{W_u},\\qquad \\Delta J_t=J_{\\theta_t}(u_t^{\\star})-J_{\\mathrm{ref}}^{\\star},$$",
  "$$\\text{accept checkpoint }t\\ \\Longleftrightarrow\\ e_t\\le\\tau_u\\ \\land\\ |\\Delta J_t|\\le\\tau_J\\ \\land\\ N(\\theta_t)\\le N_{\\max}.$$"
 ],
 "id": 2712,
 "implementation": "Add a low-frequency audit callback to neural-ODE, world-model, or differentiable-simulator training. Every $K$ optimizer updates, freeze a checkpoint, sample $M$ initial decisions using a Sobol sequence or Latin hypercube over the feasible domain, and run the same constrained optimizer from each start. Cluster final decisions using radius $\\varepsilon$; for each cluster, verify feasibility, small gradient norm $\\|\\nabla_uJ\\|\\le\\delta_g$, and positive-semidefinite Hessian eigenvalues up to tolerance $-\\delta_H$. Record the best solution $u_t^{\\star}$, cluster count $N_t$, objective gap $\\Delta J_t$, and ordinary validation loss. Keep an exponential moving average of the best audited checkpoint; stop training or restore that checkpoint if $e_t$ or $|\\Delta J_t|$ rises, even if validation loss improves. The reference can be a simulator optimum, a measured plant optimum, or the best solution at initialization. Computed quantities are forward predictions, autodiff gradients, Hessian-vector products, and multistart endpoints; only the reference tolerances and clustering radius are empirical choices. First cheap experiment: use a two-dimensional synthetic process with a unimodal true cost, train identical neural surrogates with SGD, Adam, and full-batch gradient descent, and audit every 100 steps. The predicted signature is a flat validation-loss curve alongside occasional increases in $N_t$ and $e_t$ for stochastic optimizers; checkpoint auditing should prevent accepted models with $e_t\u003e\\tau_u$ and should recover the initial optimum when training drift occurs.",
 "math_summary": "For model parameters $\\theta_t$ at training time $t$, define the decision map $\\Phi(\\theta_t)=\\{u: \\nabla_uJ_{\\theta_t}(u)=0,\\ \\nabla_u^2J_{\\theta_t}(u)\\succeq0\\}$, estimated by clustering endpoints of multistart constrained optimization. Let $u_t^{\\star}$ be the best feasible member of this set, $u_0^{\\star}$ the reference optimum, and $e_t=\\|u_t^{\\star}-u_0^{\\star}\\|$. Let $L_t$ be validation prediction loss. The mechanism is that $L_t$ can remain flat while $e_t$ and the number $N_t=|\\Phi(\\theta_t)|$ increase because stochastic gradient updates move through observationally equivalent models. Use a stopping rule $e_t\\le\\tau_u$, $|J_{\\theta_t}(u_t^{\\star})-J^\\star|\\le\\tau_J$, and $N_t\\le N_{\\max}$ rather than validation loss alone.",
 "math_tags": [
  "optimization",
  "dynamical-systems",
  "control-theory",
  "statistics"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics",
  "world-model",
  "loss"
 ],
 "paper": {
  "arxiv_id": "2608.23885",
  "arxiv_url": "https://arxiv.org/abs/2608.23885",
  "summary_what_math_gives_to_ml": "The paper identifies a practically important failure mechanism: low prediction error on historical trajectories does not preserve the economically relevant optimum of a dynamical process. Neural surrogate models can create multiple phantom optima, and stochastic gradient training can move from weights that recover the correct optimum to observationally similar weights with poor decision performance. The strongest transfer to neural networks is decision-oriented training and validation: explicitly test and regularize the surrogate's optimal-control landscape rather than relying only on trajectory fit.",
  "title": "A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 8,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "accuracy",
  "generalization"
 ],
 "title": "Phantom-Optimum Audit and Optimizer Drift Monitor",
 "url": "https://synthcore.org/idea/2712/phantom-optimum-audit-and-optimizer-drift-monitor",
 "verification": {
  "benchmark": {
   "track": "tabular",
   "model": "mlp_tiny",
   "n_seeds": 8,
   "metric_direction": "lower is better",
   "comparison": {
    "baseline_mean": 7.931941092014313,
    "idea_mean": 154.66105365753174,
    "delta_mean": 146.72911256551743,
    "per_seed_diffs": [
     115.00612020492554,
     153.04002380371094,
     98.23992824554443,
     139.6900405883789,
     170.35116958618164,
     117.17324161529541,
     170.9763307571411,
     209.35604572296143
    ],
    "idea_wins": 0,
    "n_pairs": 8,
    "p_value": 0.0081,
    "mde": 30.622178816214802,
    "mde_rel_pct": 386.0616015800278,
    "verdict": "idea worse (significant)",
    "system_worked": false
   },
   "baseline": {
    "best_cfg": {
     "lr": 0.01
    },
    "sweep": [
     {
      "cfg": {
       "lr": 0.001
      },
      "mean": 18.263182520866394
     },
     {
      "cfg": {
       "lr": 0.003
      },
      "mean": 13.55293893814087
     },
     {
      "cfg": {
       "lr": 0.01
      },
      "mean": 7.931941092014313
     }
    ],
    "full": {
     "mean": 7.931941092014313,
     "std": 0.8101161551333251,
     "per_seed": [
      7.2100510597229,
      8.814193725585938,
      7.362153053283691,
      7.630332946777344,
      6.520580291748047,
      8.472823143005371,
      8.71557903289795,
      8.729815483093262
     ],
     "n": 8
    }
   },
   "idea": {
    "mean": 154.66105365753174,
    "std": 0,
    "per_seed": [
     122.21617126464844,
     161.85421752929688,
     105.60208129882812,
     147.32037353515625,
     176.8717498779297,
     125.64606475830078,
     179.69190979003906,
     218.0858612060547
    ],
    "n": 0
   },
   "mechanism_signature": {
    "description": "trained-model checkpoint audit signature",
    "signature": {
     "confirmed": false,
     "mean_max_drift": 4.073530770838261,
     "mean_val_range": 211.3817970752716,
     "observed": [
      {
       "audits": 20,
       "max_drift": 3.82235586643219,
       "val_range": 218.36089038848877
      },
      {
       "audits": 20,
       "max_drift": 3.554457426071167,
       "val_range": 212.48634433746338
      },
      {
       "audits": 20,
       "max_drift": 5,
       "val_range": 216.55184841156006
      },
      {
       "audits": 20,
       "max_drift": 5,
       "val_range": 200.55087280273438
      },
      {
       "audits": 20,
       "max_drift": 3.406560003757477,
       "val_range": 216.53733348846436
      },
      {
       "audits": 20,
       "max_drift": 3.6925817728042603,
       "val_range": 209.86744689941406
      },
      {
       "audits": 20,
       "max_drift": 3.112291097640991,
       "val_range": 202.15625095367432
      },
      {
       "audits": 20,
       "max_drift": 5,
       "val_range": 214.5433893206787
      }
     ],
     "predicted": "validation loss can be flat while audited decision drift is nonzero"
    }
   },
   "custom_track": null
  },
  "mechanism_confirmed": false,
  "peer_reviewed": false,
  "practical_verdict": "harms",
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 8,
   "verdict": "Built a multistart bounded-optimization audit that clusters feasible stationary endpoints, counts local optima, tracks best decision drift, and applies an audit-aware checkpoint rule. The checks confirmed the predicted stationary-point transition at a=2, approximately linear decision drift with bias, and inverse-curvature drift scaling. In the synthetic trajectory, validation-only selection chose the final checkpoint with e=0.0413, while the audit selected t=4 with e=0.00980; this demonstrates the mechanism, but not yet on a trained neural surrogate.",
   "metrics": {
    "baseline": "Validation-only endpoint: validation_loss=0.0, decision drift e=0.04131, N=2",
    "idea": "Audit-selected t=4: validation_loss=7.78e-05, decision drift e=0.00980, N=2; count sweep exact across tested a values; drift linearization maximum absolute error=0.00296; curvature-slope relative errors=1.4%, 2.6%, and 6.8%"
   },
   "how_to_run": "python3 phantom_audit.py",
   "files": [
    "phantom_audit.py",
    "results.json"
   ],
   "limitations": "The experiment uses an analytically designed one-dimensional quartic surrogate and a hand-parameterized bias trajectory rather than SGD-trained neural networks, neural ODEs, or a real simulator. It does not test computational overhead, checkpoint rollback during actual training, noisy validation data, higher-dimensional constrained optimization, or Hessian-vector-product implementations."
  },
  "status": "failed_benchmark",
  "status_label": "Failed on benchmark",
  "updated_at": "2026-09-01T21:34:31",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}