{
 "artifacts": [
  {
   "name": "bench_experiment.py",
   "url": "https://synthcore.org/code/36/bench_experiment.py"
  },
  {
   "name": "bench_report.json",
   "url": "https://synthcore.org/code/36/bench_report.json"
  },
  {
   "name": "experiment.py",
   "url": "https://synthcore.org/code/36/experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/36/report.md"
  },
  {
   "name": "report_bench_2026-08-30T135854.md",
   "url": "https://synthcore.org/code/36/report_bench_2026-08-30T135854.md"
  },
  {
   "name": "report_bench_2026-08-30T153004.md",
   "url": "https://synthcore.org/code/36/report_bench_2026-08-30T153004.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/36/results.json"
  }
 ],
 "category": "optimization",
 "description": "Add an eigenvalue floor and rotation safeguard to DFP rather than trusting positive curvature and strong-Wolfe acceptance to maintain a useful inverse Hessian. The optimizer applies the ordinary DFP update when its spectrum is healthy, but damps or resets the update when the smallest inverse-Hessian eigenvalue collapses or the principal eigenspaces rotate too far between successive steps.",
 "download_zip": "https://synthcore.org/download/36",
 "formulas_latex": [
  "$$d_k=-H_k g_k,\\qquad s_k=\\alpha_k d_k,\\qquad x_{k+1}=x_k+s_k,\\qquad y_k=g_{k+1}-g_k$$",
  "$$H_{k+1}^{\\mathrm{DFP}}=H_k-\\frac{H_k y_k y_k^{T}H_k}{y_k^{T}H_k y_k}+\\frac{s_k s_k^{T}}{s_k^{T}y_k}$$",
  "$$f(x_{k+1})\\le f(x_k)+c_1g_k^Ts_k,\\qquad \\left|g_{k+1}^Ts_k\\right|\\le c_2\\left|g_k^Ts_k\\right|$$",
  "$$H_{k+1}=V\\,\\operatorname{diag}\\!\\left(\\max(\\lambda_i,\\varepsilon)\\right)V^T\\quad\\text{if }\\lambda_{\\min}(H_{k+1}^{\\mathrm{DFP}})\u003c\\varepsilon\\ \\text{or}\\ \\lVert P_{k+1}-P_k\\rVert_2\u003e\\tau$$"
 ],
 "id": 39,
 "implementation": "Integrate this into a limited-memory or blockwise DFP optimizer, because a dense full-network inverse Hessian is infeasible. Partition parameters into blocks of 256 to 2048 parameters, or use a low-rank representation H_k = gamma_k I + U U^T, and apply the safeguard independently to each block after every accepted step. First compute g_k, choose d_k = -H_k g_k, and run a line search satisfying the Armijo and strong-Wolfe tests. Store s_k = x_{k+1} - x_k and y_k = g_{k+1} - g_k. Reject the quasi-Newton update when s_k^T y_k is less than or equal to delta times the product of the norms of s_k and y_k. Otherwise form the DFP update using a dense block or limited-memory representation. Estimate the smallest and largest eigenvalues with 3 to 5 Lanczos iterations. Track P_k, the projector onto the r smallest estimated eigenvectors, and compute q_k = ||P_k - P_{k-1}||_2. If lambda_min is below epsilon, q_k exceeds tau, or the condition estimate exceeds kappa_max, use H_{k+1} = (1-rho)H_k + rho H_{k+1}^{DFP}, backtracking on rho, and clip eigenvalues to [epsilon, kappa_max epsilon]. If clipping is expensive, reset the block to gamma I with gamma = (s_k^T y_k)/(y_k^T y_k). The paper determines the failure mechanism; epsilon, tau, kappa_max, rho, and delta are empirical hyperparameters. First test on a two-layer MLP with 1,000 to 10,000 parameters on MNIST or Fashion-MNIST, comparing SGD, BFGS, ordinary DFP, and safeguarded DFP at equal gradient evaluations. Also test a two-dimensional strongly convex synthetic objective whose Hessian lies between (1/2)I and (3/2)I. Success means a lower gradient-norm plateau, bounded block condition numbers, fewer line-search failures, and lower loss at equal function and gradient evaluations.",
 "math_summary": "The paper's classical DFP iteration uses gradient g_k = grad f(x_k), search direction d_k = -H_k g_k, step s_k = alpha_k d_k, new gradient g_{k+1} = grad f(x_{k+1}), and secant vector y_k = g_{k+1} - g_k. Its inverse-Hessian update is H_{k+1} = H_k - (H_k y_k y_k^T H_k)/(y_k^T H_k y_k) + (s_k s_k^T)/(s_k^T y_k). Under s_k^T y_k \u003e 0, this update can remain positive definite, but the paper constructs a uniformly strongly convex function f with (1/2)I less than or equal to grad^2 f(x) less than or equal to (3/2)I where the smaller eigenvalue of H_k tends to zero and the associated eigenvectors rotate with unbounded total angle, even though the strong-Wolfe conditions hold. The conditions are f(x_{k+1}) less than or equal to f(x_k) + c_1 g_k^T s_k and absolute value of g_{k+1}^T s_k less than or equal to c_2 times absolute value of g_k^T s_k. The transferred safeguard enforces lambda_min(H_{k+1}) greater than or equal to epsilon and limits eigenspace motion using sin(theta_k) = ||P_k - P_{k-1}||_2, where P_k is the projector onto the eigenvectors associated with the smallest r eigenvalues. A practical implementation estimates these spectral quantities with Lanczos or randomized power iteration.",
 "math_tags": [
  "optimization",
  "numerical-analysis",
  "linear-algebra",
  "dynamical-systems"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.21708",
  "arxiv_url": "https://arxiv.org/abs/2608.21708",
  "summary_what_math_gives_to_ml": "The paper gives a constructive failure mode for classical DFP that survives uniformly bounded objective curvature and standard strong-Wolfe line searches: the inverse-Hessian approximation can lose its smallest eigenvalue while its eigenspaces rotate indefinitely. This is valuable for neural-network quasi-Newton training because line-search acceptance and positive-definite updates alone do not prevent an optimizer from becoming effectively singular in changing gradient directions. The direct transfer is a spectrally safeguarded DFP optimizer that monitors inverse-Hessian conditioning and eigenspace rotation, then damps, clips, or resets problematic updates. The construction also supplies an adversarial benchmark: a smooth, globally well-conditioned two-dimensional objective on which Wolfe-compliant DFP fails despite benign true curvature.",
  "title": "A counterexample to global convergence of classical DFP under the standard strong Wolfe conditions",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 5,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "accuracy"
 ],
 "title": "Spectrally safeguarded DFP",
 "url": "https://synthcore.org/idea/39/spectrally-safeguarded-dfp",
 "verification": {
  "benchmark": {
   "track": "tabular",
   "model": "mlp_tiny",
   "n_seeds": 8,
   "metric_direction": "lower is better",
   "comparison": {
    "baseline_mean": 0,
    "idea_mean": 13.662639737129211,
    "delta_mean": -21.711756348609924,
    "per_seed_diffs": [],
    "idea_wins": 8,
    "n_pairs": 8,
    "p_value": 0.0081,
    "mde": 0,
    "mde_rel_pct": null,
    "verdict": "idea better (significant)",
    "system_worked": true
   },
   "baseline": {
    "best_cfg": {
     "epochs": 8,
     "lr": 0.01
    },
    "sweep": null,
    "full": {
     "mean": 0,
     "std": 0,
     "per_seed": null,
     "n": 0
    }
   },
   "idea": {
    "mean": 13.662639737129211,
    "std": 1.2255122413284667,
    "per_seed": [
     13.658586502075195,
     10.993560791015625,
     14.850048065185547,
     14.97375202178955,
     12.930068016052246,
     13.51315689086914,
     13.623165130615234,
     14.758780479431152
    ],
    "n": 0
   },
   "mechanism_signature": {
    "confirmed": true,
    "observed_max_rotation": 0.9999999997777941,
    "observed_min_eig_min": 0.00015548363137000883,
    "predicted_floor": 0.0001,
    "safeguards_total": 374
   },
   "custom_track": null
  },
  "mechanism_confirmed": true,
  "peer_reviewed": false,
  "practical_verdict": "helps",
  "stage1_mechanism_check": {
   "worked": false,
   "confidence": 9,
   "verdict": "Built a dense small-network spectrally safeguarded DFP optimizer with eigenvalue flooring, condition control, and smallest-eigenspace rotation monitoring. The numerical checks passed: DFP satisfied H_new y=s with error 5.9e-15, and the safeguard repaired a 1e-8 eigenvalue to the 1e-3 floor. However, at equal 100 gradient evaluations on the fixed synthetic classification dataset, ordinary DFP reached loss 1.77e-5 while safeguarded DFP reached 1.52e-2 and triggered 30 safeguards, so the proposed safeguard did not produce a win in this MVP.",
   "metrics": {
    "baseline": "Ordinary DFP: final loss 1.77e-5, gradient norm 5.19e-4, minimum observed H eigenvalue 1.96e-2, 100 gradient evaluations. SGD: final loss 0.6931, gradient norm 0.1505, 100 evaluations.",
    "idea": "Safeguarded DFP: final loss 1.52e-2, gradient norm 1.39e-2, minimum observed H eigenvalue 3.11e-2, 30 safeguards, 100 gradient evaluations. Math check: secant error 5.86e-15; safeguarded minimum eigenvalue 1e-3."
   },
   "how_to_run": "python3 experiment.py",
   "files": [
    "experiment.py",
    "results.json"
   ],
   "limitations": "Only one fixed synthetic classification dataset, one seed, one small dense model, and one short CPU run were tested. The implementation uses full dense eigendecompositions rather than blockwise or limited-memory spectral estimates, and its line search is Armijo-only rather than a complete strong-Wolfe search; no MNIST, 2D paper counterexample, FLOP accounting, or multi-seed statistics were evaluated."
  },
  "status": "beats_baseline",
  "status_label": "Beats tuned baseline",
  "updated_at": "2026-09-02T12:30:10",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": true,
    "tested": true
   },
   "practical_benchmark": {
    "beats_baseline": true,
    "tested": true,
    "verdict": "helps"
   },
   "toy_mechanism_gate": {
    "confirmed": false,
    "tested": true
   }
  }
 }
}