✗ Failed on benchmark 2026

Recycled-curvature proximal optimizer

Implementation & benchmark of arXiv:2607.22895 — Curvature Recycling Douglas-Rachford Splitting: Transported Quasi-Newton Models for Expensive Smooth Proximal Subproblems

Usefulness7/10

Difficulty5/10

Novelty7/10

Optimization Speedup Stability Memory Optimizer Regularization Training dynamics Optimization Convex analysis Numerical analysis

Source paper: Curvature Recycling Douglas-Rachford Splitting: Transported Quasi-Newton Models for Expensive Smooth Proximal Subproblems arXiv:2607.22895 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace independently restarted proximal-gradient or quasi-Newton solves for a composite neural objective with a curvature-recycling Douglas–Rachford loop. The previous proximal state, residual, and limited-memory BFGS curvature pairs are transported to the next proximal center, reducing expensive loss and gradient evaluations while retaining the cheap nonsmooth proximal operation.

Formulas

$$P_{\gamma f}z:=\arg\min_x\left\{\gamma f(x)+\frac{1}{2}\lVert x-z\rVert^2\right\},\qquad r_z(x):=\gamma\nabla f(x)+x-z.$$

$$r_{z_{\mathrm{new}}}(x)=r_{z_{\mathrm{old}}}(x)+z_{\mathrm{old}}-z_{\mathrm{new}}.$$

$$p_k=P_{\gamma f}z_k,\qquad y_k^g=P_{\gamma g}(2p_k-z_k),\qquad z_{k+1}=z_k+2\alpha(y_k^g-p_k).$$

$$H_{k+1}=(I-\rho_k s_kq_k^T)H_k(I-\rho_k q_ks_k^T)+\rho_k s_ks_k^T,\quad \rho_k=(q_k^Ts_k)^{-1},\quad d_k=-H_kr_k.$$

Mathematical statement

The smooth proximal map is P_gamma_f(z) = argmin_x { gamma f(x) + 0.5 ||x - z||^2 }, where f is the expensive smooth term, gamma > 0 is the proximal scale, and z is the current Douglas–Rachford center. Its optimality residual is r_z(x) = gamma grad f(x) + x - z. If the old center is z_old and the new center is z_new, then the same point x has exactly transported residual r_{z_new}(x) = r_{z_old}(x) + z_old - z_new. Therefore an old approximate proximal state can be reused before computing a new expensive gradient. Maintain an inverse-Hessian approximation H with BFGS pairs s = x_new - x_old and q = r_new - r_old. When q transpose s is positive, update H by the inverse BFGS formula and use d = -H r as the quasi-Newton direction.

Implementation notes

1. Integration point: use this as the inner solver for a composite neural objective F(theta) = f(theta) + g(theta), where f is an expensive differentiable training loss and g has a cheap proximal map. Suitable targets are group-lasso regularization on attention heads or MLP blocks, elementwise L1 pruning, or a small implicit neural layer with an expensive forward solve. Replace the optimizer step that approximately solves the smooth proximal subproblem; ordinary forward inference is unchanged.

2. Pseudocode: initialize z = theta, p = theta, and an empty L-BFGS history. At each outer iteration, compute the new center z_new. Transport the old residual with r = r + z_old - z_new and use p as the warm start. For j = 1,...,J, compute d = -H(r) using the L-BFGS two-loop recursion, choose a clipped step length a, set p_trial = p + a d, evaluate the expensive gradient at p_trial, and form r_trial = gamma grad f(p_trial) + p_trial - z_new. Store s = p_trial - p and q = r_trial - r if q transpose s exceeds a curvature threshold, then replace p and r. Apply the cheap proximal map y = prox_{gamma g}(2p - z_new), and update z = z_new + 2 alpha (y - p).

3. Compute from the mathematics: the residual, exact center-transport correction, Douglas–Rachford update, and BFGS secant update. Estimate empirically the inner iteration count J, damping, line-search constants, history size, and curvature retention after center changes. Use separate small L-BFGS histories per parameter group rather than a dense matrix.

4. First experiment: train a ViT-Tiny or two-layer MLP on CIFAR-10 with L1 or group-lasso regularization and an intentionally expensive auxiliary forward loss. Compare restarted proximal gradient, restarted L-BFGS proximal solves, and recycled L-BFGS at matched outer updates. Measure expensive gradient calls, wall-clock time, sparsity, residual norm, training loss, and validation accuracy. The target signal is at least 25 percent fewer expensive gradient evaluations at equal final accuracy without increased instability.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a self-contained quadratic composite-proximal experiment with exact center-residual transport, secant reuse, inverse-BFGS updating, and frozen modeled rollouts. The mechanism checks passed: transport error was 2.48e-16, secant identity error was 1.42e-14, the observed secant curvature ratio stayed above the predicted lower bound 1, and the H=I stability sweep matched the predicted contractive interval 0 < gamma*lambda < 2. The recycled method used 31 versus 90 expensive gradient calls, a 65.6% reduction, but had substantially worse final residual (0.109 versus 0.00614) and was slower on this tiny dense problem, so the claimed end-to-end win was not demonstrated.

Agent confidence: 8/10

Baseline: 90 gradient calls; final residual 0.00614; mean residual 0.00780; 0.00219 s
Idea: 31 gradient calls; 65.6% fewer; final residual 0.10894; mean residual 0.18040; 0.13316 s
Math: transport error 2.48e-16; secant identity error 1.42e-14; minimum secant ratio 1.1685 versus bound 1; stability boundary consistent with gamma*lambda=2

Limitations: Only a small convex quadratic toy problem was tested, not a neural network, CIFAR-10, group sparsity, noisy gradients, wall-clock expensive forward loss, or larger-scale L-BFGS memory behavior. The recycled predictor used a dense inverse matrix in the toy implementation and therefore its wall-clock timing is not representative of a production limited-memory implementation.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.22895) Sep 2, 2026

Significantly WORSE than the tuned baseline (+1571.9%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.4183
Idea mean: 124.025
Effect (Δ): 116.6066 (+1571.9%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±236.5%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 4 configs

Limitations:

Only the structurally matched tabular/Friedman track and mlp_tiny model were tested; vision, sequence, and dynamics tracks were not tested. The implementation uses a full flattened parameter vector rather than production per-parameter-group L-BFGS storage, and the benchmark's small minibatch regime did not create expensive-gradient savings.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented the recycled-curvature proximal optimizer as a custom training-loop intervention on the structurally matched tabular Friedman regression track with the shared mlp_tiny architecture. The full 8-seed paired protocol and shared learning-rate sweep showed the idea was significantly worse: test MSE 124.025 versus baseline 7.418, paired delta +116.607, permutation p=0.0081. The trained-model transport identity was confirmed at mean error 1.59e-6, but the promised evaluation reduction was not observed: 73 versus 72 gradient calls on average.

Baseline: Adam, best lr=0.03; 8-seed mean test MSE 7.4183, std 0.9955; sweep means lr=0.001:24.1424, lr=0.003:13.7540, lr=0.01:8.3259, lr=0.03:6.8654; final paired-run mean gradient calls 72
Comparison: delta_mean=+116.6066 (idea minus baseline), 0/8 idea wins, permutation p=0.0081, verdict=idea worse (significant)
Idea: Recycled-curvature proximal loop, best tested lr=0.03; 8-seed mean test MSE 124.0250, std 19.8886; idea trial means lr=0.003:235.4396, lr=0.01:223.3305, lr=0.03:130.3428; final paired-run mean gradient calls 73
Mechanism signature: Mean trained-model residual transport error 1.593e-6 versus predicted zero; confirmed transport arithmetic but not call reduction, with observed call-reduction fraction -0.0139

Benchmark result