✗ Failed on benchmark 2026

Minimal Negative-Curvature L-BFGS

Implementation & benchmark of arXiv:2608.29300 — Negative-Curvature-Informed L-BFGS via Minimal Secant Corrections for Finite Minimax Problems

Usefulness7/10

Difficulty5/10

Novelty7/10

Optimization Stability Speedup Accuracy Optimizer Loss Training dynamics Fine tuning Optimization Convex analysis Numerical analysis

Source paper: Negative-Curvature-Informed L-BFGS via Minimal Secant Corrections for Finite Minimax Problems arXiv:2608.29300 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Modify an L-BFGS curvature pair only when the observed secant curvature is negative. Replace the gradient-difference vector by the smallest Euclidean or inverse-metric correction that enforces positive curvature, then use the unmodified BFGS update and two-loop recursion. This avoids the computational and conditioning cost of adding a large isotropic damping term to the whole inverse-Hessian approximation.

Formulas

$$s_k=x_{k+1}-x_k,\qquad y_k=g_{k+1}-g_k,\qquad s_k^T y_k<0,$$

$$s_k^T\widetilde y_k=|s_k^T y_k|,$$

$$\widetilde y_k^{\mathrm{Euc}}=y_k+\frac{|s_k^T y_k|-s_k^T y_k}{s_k^T s_k}s_k,$$

$$\widetilde y_k^{B^{-1}}=y_k+\frac{|s_k^T y_k|-s_k^T y_k}{s_k^T B_k s_k}B_k s_k,\qquad B_{k+1}=B_k-\frac{B_k s_k s_k^T B_k}{s_k^T B_k s_k}+\frac{\widetilde y_k\widetilde y_k^T}{s_k^T\widetilde y_k}.$$

Mathematical statement

Let x_k be the parameter vector at iteration k, s_k=x_{k+1}-x_k the step, and y_k=g_{k+1}-g_k the gradient-difference vector, where g_k is the gradient of the current smooth objective. A pair has strict negative curvature when s_k^T y_k<0. The correction target is s_k^T\widetilde y_k=|s_k^T y_k|>0. The sign-flip rule uses \widetilde y_k=-y_k. The Euclidean nearest-point rule solves min_y ||y-y_k||_2^2 subject to s_k^T y=|s_k^T y_k|, giving \widetilde y_k=y_k+(|s_k^T y_k|-s_k^T y_k)s_k/(s_k^T s_k). If B_k is the positive-definite L-BFGS Hessian approximation, the B_k^{-1}-metric rule solves min_y (y-y_k)^T B_k^{-1}(y-y_k) under the same secant constraint, giving \widetilde y_k=y_k+(|s_k^T y_k|-s_k^T y_k)B_k s_k/(s_k^T B_k s_k). The standard BFGS update is then B_{k+1}=B_k-(B_k s_k s_k^T B_k)/(s_k^T B_k s_k)+(\widetilde y_k\widetilde y_k^T)/(s_k^T\widetilde y_k). Because both denominators are positive, positive definiteness is preserved when B_k is positive definite.

Implementation notes

Integrate the method into the curvature-pair insertion step of a standard L-BFGS optimizer, rather than changing the network architecture. Use it for full-batch or large-batch fine-tuning, where gradients at consecutive iterates are sufficiently comparable. Maintain the usual memory of the most recent m pairs (s_i,y_i), and use the existing two-loop recursion to compute p_k=-H_k g_k. After a line search produces x_{k+1}, compute s=x_{k+1}-x_k and y=g_{k+1}-g_k. Let c=s^T y and use a scale-aware negative-curvature test c < -epsilon_curv ||s||_2 ||y||_2, with epsilon_curv initially 1e-4. If the test is false, insert the ordinary pair. If true, choose one variant: sign flip ytilde=-y; Euclidean projection ytilde=y+(abs(c)-c)s/(s^T s); or inverse-metric projection ytilde=y+(abs(c)-c)B s/(s^T B s). The inverse-metric version requires applying the current limited-memory B operator to s; implement this through the stored L-BFGS pairs, or initially test the Euclidean version. Enforce ctilde=max(abs(c),epsilon_sec ||s||_2 ||y||_2) if floating-point noise makes the denominator too small. Insert (s,ytilde), then run the unchanged two-loop recursion. The paper-derived quantities are the curvature test, projection formulas, and positive-secant guarantee; curvature is estimated empirically from consecutive gradients, and B actions are estimated from the current L-BFGS memory. First experiment: fine-tune a 10–50M parameter Transformer or ResNet on a small dataset using a finite minimax loss over four augmentations or task heads. Compare AdamW, ordinary L-BFGS, damped L-BFGS, and all three correction rules. Track loss versus gradient evaluations and wall-clock time, fraction of corrected pairs, line-search failure rate, NaNs, and final worst-case component loss. Success means fewer rejected line searches and faster decrease of the maximum component loss than ordinary L-BFGS without a significant per-step overhead.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a self-contained Euclidean minimal negative-curvature L-BFGS prototype with two-loop recursion, curvature-pair correction, line search, and numerical verification. Across rho=0.001–0.9, the corrected secant ratio matched 1.0 and the correction norm matched the predicted 2rho scaling to about 1e-15; corrected BFGS updates remained positive definite in dimensions 3, 8, and 20. In the toy optimizer, correction reduced cumulative line-search halvings from 164 to 1 after one negative-curvature event, although both methods reached the same final loss, so a broad optimization win is not established.

Agent confidence: 8/10

Baseline: final loss -0.625; 180 steps; 1 negative pair rejected; 164 cumulative backtracking halvings; finite
Idea: final loss -0.625; 180 steps; 1 negative pair corrected; 1 cumulative backtracking halving; finite; secant ratio 1.0; correction-scaling error approximately 1e-15

Limitations: Only the Euclidean correction was implemented; inverse-metric and sign-flip variants were not tested. The optimizer experiment is a tiny 12-dimensional synthetic objective with only one observed negative-curvature pair and no wall-clock/FLOP study, stochastic-noise study, or neural-network benchmark.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.29300) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 12.4191
Idea mean: 12.4191
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched official tabular track was tested. The official baseline sweep uses its prescribed four sweep seeds, while the selected baseline and idea are evaluated on eight paired seeds. In this setup no negative-curvature pair appeared, so correction behavior, inverse-metric correction, sign-flip correction, damped L-BFGS, and performance under deliberately more nonconvex or noisy training conditions were not tested.

How to run: python3 official_stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented Minimal Negative-Curvature L-BFGS as an optimizer-only modification of the official tabular mlp_tiny training procedure, using the registered tabular track and Euclidean secant correction. The official benchmark produced identical baseline and idea test MSE: delta_mean=0.0 with permutation p=1.0; no negative-curvature pairs or corrections occurred across the paired runs. Therefore the idea did not demonstrate a significant improvement.

Baseline: Official tabular/mlp_tiny: tuned best lr=0.12, epochs=15; full 8-seed mean test MSE 12.419063, std 4.653165; baseline sweep means: lr 0.03 -> 17.872909, lr 0.06 -> 18.236159, lr 0.12 -> 7.929520 on sweep seeds.
Idea: Official tabular/mlp_tiny: best lr=0.12, epochs=15; 8-seed mean test MSE 12.419063, std 4.653165; identical per-seed results; 0 negative pairs, 0 corrected pairs; delta_mean=0.0; permutation p=1.0; verdict=no measurable effect.

Benchmark result