# Certified curvature-recycling fallback

- ID: 1298
- Canonical URL: https://synthcore.org/idea/1298/certified-curvature-recycling-fallback
- API JSON: https://synthcore.org/api/idea/1298.json
- API Markdown: https://synthcore.org/api/idea/1298.md
- Verification status: unverified
- Source: [arXiv:2607.22895](https://arxiv.org/abs/2607.22895)
- Category: optimization
- Solves: stability, speedup, accuracy
- ML areas: optimizer, training-dynamics, regularization
- Math tags: optimization, convex-analysis, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Add an acceptance gate around transported quasi-Newton steps: use recycled curvature only when it decreases the smooth proximal merit and reduces the new residual. Otherwise discard the candidate and execute a bounded number of conservative gradient steps, making curvature reuse robust to minibatch changes and stale models.

## Mathematical statement

For center z, define the smooth proximal objective phi_z(x) = gamma f(x) + 0.5 ||x - z||^2 and residual r_z(x) = grad phi_z(x) = gamma grad f(x) + x - z. Given a recycled inverse-curvature model H, form a candidate x_plus = x + a d with d = -H r_z(x). Accept it only if the sufficient-descent inequality phi_z(x_plus) <= phi_z(x) - c a r_z(x) transpose d holds and the residual contracts according to ||r_z(x_plus)|| <= eta ||r_z(x)||, where c is in (0,1) and eta is less than 1. If either test fails, use safe gradient steps x <- x - tau r_z(x). When grad f is L_f-Lipschitz, grad phi_z is Lipschitz with constant 1 + gamma L_f, so tau <= 1/(1 + gamma L_f) is a conservative step-size bound.

## Key formulas

- $$\phi_z(x):=\gamma f(x)+\frac12\lVert x-z\rVert^2,\qquad r_z(x)=\nabla\phi_z(x)=\gamma\nabla f(x)+x-z.$$
- $$x^+=x+a d,\qquad d=-H r_z(x),\qquad \phi_z(x^+)\leq\phi_z(x)-c a\,r_z(x)^Td.$$
- $$\lVert r_z(x^+)\rVert\leq\eta\lVert r_z(x)\rVert,\qquad 0<\eta<1;\quad\text{otherwise }x\leftarrow x-\tau r_z(x),\quad 0<\tau\leq(1+\gamma L_f)^{-1}.$$
- $$r_{z_{\mathrm{new}}}(x)=r_{z_{\mathrm{old}}}(x)+z_{\mathrm{old}}-z_{\mathrm{new}}.$$

## Implementation notes

1. Integration point: wrap the inner optimizer used for each proximal update of neural parameters. Evaluate the gate after a transported L-BFGS, SR1, or diagonal quasi-Newton candidate and before committing that step. This is especially useful for minibatch training, where both the proximal center and sampled loss change between iterations.

2. Pseudocode: input center z, warm-start x, transported residual r, inverse model H, proximal scale gamma, and an estimate L_f. Set c = 1e-4, eta = 0.9, and tau = 0.5/(1 + gamma L_f). Compute d = -H r and initially set a = 1. Evaluate x_plus = x + a d, the expensive loss f(x_plus), and gradient grad f(x_plus). Form r_plus = gamma grad f(x_plus) + x_plus - z. Accept only if phi_z(x_plus) <= phi_z(x) - c a r transpose d and norm(r_plus) <= eta norm(r). If rejected, discard the candidate and execute m = 2 or 3 fallback steps: evaluate the gradient at the current x, update x <- x - tau r_z(x), and recompute the residual. Update the curvature history only for pairs satisfying q transpose s > epsilon norm(s) norm(q).

3. Compute from the mathematics: the proximal merit, exact residual, descent test, residual contraction test, and conservative gradient-step bound. Estimate L_f from secant pairs using L_f <- max(L_f, norm(q_f)/norm(s)), multiplied by a safety factor such as 1.2; clip the estimate to prevent one anomalous minibatch from making all fallback steps too small. Record candidate acceptance and rejection statistics.

4. First experiment: use a six-layer MLP on MNIST or CIFAR-10 with group-lasso regularization and deliberately vary minibatches between proximal solves. Compare recycled L-BFGS without a gate, the certified gate, and proximal SGD. Track loss versus expensive gradient calls, accepted-step fraction, residual contraction, gradient spikes, and validation accuracy. The target signal is at least 20 percent fewer expensive gradients than proximal SGD at equal accuracy, while eliminating loss spikes caused by ungated curvature reuse.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.