# Recycled-curvature proximal optimizer

- ID: 1297
- Canonical URL: https://synthcore.org/idea/1297/recycled-curvature-proximal-optimizer
- API JSON: https://synthcore.org/api/idea/1297.json
- API Markdown: https://synthcore.org/api/idea/1297.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.22895](https://arxiv.org/abs/2607.22895)
- Category: optimization
- Solves: speedup, stability, memory
- ML areas: optimizer, regularization, training-dynamics
- Math tags: optimization, convex-analysis, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace independently restarted proximal-gradient or quasi-Newton solves for a composite neural objective with a curvature-recycling Douglas–Rachford loop. The previous proximal state, residual, and limited-memory BFGS curvature pairs are transported to the next proximal center, reducing expensive loss and gradient evaluations while retaining the cheap nonsmooth proximal operation.

## Mathematical statement

The smooth proximal map is P_gamma_f(z) = argmin_x { gamma f(x) + 0.5 ||x - z||^2 }, where f is the expensive smooth term, gamma > 0 is the proximal scale, and z is the current Douglas–Rachford center. Its optimality residual is r_z(x) = gamma grad f(x) + x - z. If the old center is z_old and the new center is z_new, then the same point x has exactly transported residual r_{z_new}(x) = r_{z_old}(x) + z_old - z_new. Therefore an old approximate proximal state can be reused before computing a new expensive gradient. Maintain an inverse-Hessian approximation H with BFGS pairs s = x_new - x_old and q = r_new - r_old. When q transpose s is positive, update H by the inverse BFGS formula and use d = -H r as the quasi-Newton direction.

## Key formulas

- $$P_{\gamma f}z:=\arg\min_x\left\{\gamma f(x)+\frac{1}{2}\lVert x-z\rVert^2\right\},\qquad r_z(x):=\gamma\nabla f(x)+x-z.$$
- $$r_{z_{\mathrm{new}}}(x)=r_{z_{\mathrm{old}}}(x)+z_{\mathrm{old}}-z_{\mathrm{new}}.$$
- $$p_k=P_{\gamma f}z_k,\qquad y_k^g=P_{\gamma g}(2p_k-z_k),\qquad z_{k+1}=z_k+2\alpha(y_k^g-p_k).$$
- $$H_{k+1}=(I-\rho_k s_kq_k^T)H_k(I-\rho_k q_ks_k^T)+\rho_k s_ks_k^T,\quad \rho_k=(q_k^Ts_k)^{-1},\quad d_k=-H_kr_k.$$

## Implementation notes

1. Integration point: use this as the inner solver for a composite neural objective F(theta) = f(theta) + g(theta), where f is an expensive differentiable training loss and g has a cheap proximal map. Suitable targets are group-lasso regularization on attention heads or MLP blocks, elementwise L1 pruning, or a small implicit neural layer with an expensive forward solve. Replace the optimizer step that approximately solves the smooth proximal subproblem; ordinary forward inference is unchanged.

2. Pseudocode: initialize z = theta, p = theta, and an empty L-BFGS history. At each outer iteration, compute the new center z_new. Transport the old residual with r = r + z_old - z_new and use p as the warm start. For j = 1,...,J, compute d = -H(r) using the L-BFGS two-loop recursion, choose a clipped step length a, set p_trial = p + a d, evaluate the expensive gradient at p_trial, and form r_trial = gamma grad f(p_trial) + p_trial - z_new. Store s = p_trial - p and q = r_trial - r if q transpose s exceeds a curvature threshold, then replace p and r. Apply the cheap proximal map y = prox_{gamma g}(2p - z_new), and update z = z_new + 2 alpha (y - p).

3. Compute from the mathematics: the residual, exact center-transport correction, Douglas–Rachford update, and BFGS secant update. Estimate empirically the inner iteration count J, damping, line-search constants, history size, and curvature retention after center changes. Use separate small L-BFGS histories per parameter group rather than a dense matrix.

4. First experiment: train a ViT-Tiny or two-layer MLP on CIFAR-10 with L1 or group-lasso regularization and an intentionally expensive auxiliary forward loss. Compare restarted proximal gradient, restarted L-BFGS proximal solves, and recycled L-BFGS at matched outer updates. Measure expensive gradient calls, wall-clock time, sparsity, residual norm, training loss, and validation accuracy. The target signal is at least 25 percent fewer expensive gradient evaluations at equal final accuracy without increased instability.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a self-contained quadratic composite-proximal experiment with exact center-residual transport, secant reuse, inverse-BFGS updating, and frozen modeled rollouts. The mechanism checks passed: transport error was 2.48e-16, secant identity error was 1.42e-14, the observed secant curvature ratio stayed above the predicted lower bound 1, and the H=I stability sweep matched the predicted contractive interval 0 < gamma*lambda < 2. The recycled method used 31 versus 90 expensive gradient calls, a 65.6% reduction, but had substantially worse final residual (0.109 versus 0.00614) and was slower on this tiny dense problem, so the claimed end-to-end win was not demonstrated.

### Mechanism check

- Verdict: Built a self-contained quadratic composite-proximal experiment with exact center-residual transport, secant reuse, inverse-BFGS updating, and frozen modeled rollouts. The mechanism checks passed: transport error was 2.48e-16, secant identity error was 1.42e-14, the observed secant curvature ratio stayed above the predicted lower bound 1, and the H=I stability sweep matched the predicted contractive interval 0 < gamma*lambda < 2. The recycled method used 31 versus 90 expensive gradient calls, a 65.6% reduction, but had substantially worse final residual (0.109 versus 0.00614) and was slower on this tiny dense problem, so the claimed end-to-end win was not demonstrated.
- Confidence: 8/10
- Limitations: Only a small convex quadratic toy problem was tested, not a neural network, CIFAR-10, group sparsity, noisy gradients, wall-clock expensive forward loss, or larger-scale L-BFGS memory behavior. The recycled predictor used a dense inverse matrix in the toy implementation and therefore its wall-clock timing is not representative of a production limited-memory implementation.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/515/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/515/bench_report.json)
- [experiment.py](https://synthcore.org/code/515/experiment.py)
- [report.md](https://synthcore.org/code/515/report.md)
- [report_bench_2026-08-31T163740.md](https://synthcore.org/code/515/report_bench_2026-08-31T163740.md)
- [results.json](https://synthcore.org/code/515/results.json)
- [Download all files as ZIP](https://synthcore.org/download/515)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.