# Adaptive Proximal Quasi-Newton Training

- ID: 2394
- Canonical URL: https://synthcore.org/idea/2394/adaptive-proximal-quasi-newton-training
- API JSON: https://synthcore.org/api/idea/2394.json
- API Markdown: https://synthcore.org/api/idea/2394.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.16280](https://arxiv.org/abs/2608.16280)
- Category: optimizer
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics, regularization
- Math tags: optimization, linear-algebra, dynamical-systems, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace the raw gradient step for a neural-network parameter block with a proximal quasi-Newton step, using the proximal operator to enforce nonsmooth constraints or structured regularization and an adaptive linesearch that enlarges the stepsize after several successful iterations. The method should permit much larger steps than conservative monotone backtracking while retaining a residual-decrease safeguard near unstable regions.

## Mathematical statement

Let $w\in\mathbb{R}^{d}$ be a trainable parameter block, $F(w)$ the smooth minibatch objective, $R(w)$ a proper proximable nonsmooth penalty, $g_t=\nabla F(w_t)$, $H_t\succ0$ an inverse-curvature approximation, and $\eta_t>0$ a stepsize. The proximal quasi-Newton update is $w_{t+1}=\operatorname{prox}^{H_t}_{\eta_tR}(w_t-\eta_tH_tg_t)$, where $\operatorname{prox}^{H}_{\eta R}(z)=\arg\min_v[R(v)+(2\eta)^{-1}(v-z)^TH^{-1}(v-z)]$. Accept a trial step only when a residual or objective merit decreases sufficiently; after $q$ consecutive accepted iterations enlarge $\eta_t$ by $\gamma_{\uparrow}>1$, while rejected steps shrink it by $\gamma_{\downarrow}\in(0,1)$. For a locally quadratic objective with Hessian eigenvalues bounded above, the linearized iteration is stable when $0<\eta_t<2/\lambda_{\max}(H_t^{1/2}\nabla^2F(w_*)H_t^{1/2})$.

## Key formulas

- $$w_{t+1}=\operatorname{prox}^{H_t}_{\eta_tR}\left(w_t-\eta_tH_t\nabla F(w_t)\right),\qquad \operatorname{prox}^{H}_{\eta R}(z)=\arg\min_v\left\{R(v)+\frac{1}{2\eta}(v-z)^TH^{-1}(v-z)\right\}.$$
- $$s_t=w_{t+1}-w_t,\quad y_t=\nabla F(w_{t+1})-\nabla F(w_t),\quad H_{t+1}=\left(I-\frac{s_ty_t^T}{y_t^Ts_t}\right)H_t\left(I-\frac{y_ts_t^T}{y_t^Ts_t}\right)+\frac{s_ts_t^T}{y_t^Ts_t}.$$
- $$\eta_{t+1}=\begin{cases}\gamma_{\uparrow}\eta_t,&\text{if the last }q\text{ trials were accepted},\\\gamma_{\downarrow}\eta_t,&\text{if the trial merit condition fails},\\\eta_t,&\text{otherwise},\end{cases}\qquad 0<\gamma_{\downarrow}<1<\gamma_{\uparrow}.$$
- $$0<\eta_t<\eta_c\equiv\frac{2}{\lambda_{\max}\left(H_t^{1/2}\nabla^2F(w_*)H_t^{1/2}\right)}.$$

## Implementation notes

1. Integration point: use this optimizer on one or more sensitive parameter blocks of a small MLP, transformer feed-forward layer, or low-rank adapter. Set $R(w)=\lambda\|w\|_1$ for a sparse adapter, use an indicator function for box-constrained weights, or set $R(w)=0$ for a pure quasi-Newton comparison. Do not form $H_t$ densely; begin with diagonal scaling, limited-memory BFGS pairs, or a compact two-loop inverse-Hessian application. 2. Pseudocode: compute $g=\operatorname{backward}(F,w)$; propose $z=w-\eta H g$; compute $w^{trial}=\operatorname{prox}^{H}_{\eta R}(z)$; evaluate a fresh or cached minibatch merit $M$; accept if $M(w^{trial})\leq M(w)+c\,g^T(w^{trial}-w)$ or if a fixed-point residual decreases; on acceptance update $(s,y)$ and increase the success counter; on rejection restore $w$, shrink $\eta$, and clear unstable curvature pairs. Enlarge the stepsize only after $q=3$ successes. 3. Computed versus estimated: gradients and objective values are computed by autodiff; curvature is estimated from successive gradients; $\lambda_{\max}$ is estimated with 5--10 Hessian-vector power iterations, not explicitly computed. 4. First cheap experiment: train a two-layer ReLU MLP on MNIST or CIFAR-10 using SGD, AdamW, fixed-step proximal gradient, and this method with identical minibatches. Sweep the initial $\eta$ and record accepted/rejected trials, loss, and gradient norm. The prediction is a sharp divergence or oscillation boundary near $\eta_c=2/\lambda_{\max}(H^{1/2}\nabla^2F H^{1/2})$ on a fixed small-batch quadratic or nearly quadratic network; the measured largest stable stepsize should agree with the power-iteration estimate within approximately 20%, while adaptive enlargement should settle within 10--20% below that boundary.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a deterministic proximal quasi-Newton-style optimizer and verified the quadratic stability mechanism. The predicted stability boundary was eta_c=0.08889, while the observed binary-search boundary was 0.08915 (0.3% error); the predicted contraction factor 0.97556 matched 0.97343 observed. Adaptive enlargement produced 157 accepted and 23 rejected trials and reached the same sparse-regression objective as fixed proximal gradient, although its late stepsizes sometimes exceeded eta_c, so the claimed conservative settling behavior was not fully demonstrated.

### Mechanism check

- Verdict: Built a deterministic proximal quasi-Newton-style optimizer and verified the quadratic stability mechanism. The predicted stability boundary was eta_c=0.08889, while the observed binary-search boundary was 0.08915 (0.3% error); the predicted contraction factor 0.97556 matched 0.97343 observed. Adaptive enlargement produced 157 accepted and 23 rejected trials and reached the same sparse-regression objective as fixed proximal gradient, although its late stepsizes sometimes exceeded eta_c, so the claimed conservative settling behavior was not fully demonstrated.
- Confidence: 8/10
- Limitations: This was a toy diagonal quadratic and small full-batch sparse linear regression, not an MLP or image benchmark. The implementation uses diagonal secant curvature rather than limited-memory dense quasi-Newton pairs, does not measure wall-clock/FLOP speedup, and the adaptive stepsize was not consistently below the theoretical boundary near convergence.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/939/bench_report.json)
- [bench_stage2.py](https://synthcore.org/code/939/bench_stage2.py)
- [experiment.py](https://synthcore.org/code/939/experiment.py)
- [report.md](https://synthcore.org/code/939/report.md)
- [report_bench_2026-09-01T162004.md](https://synthcore.org/code/939/report_bench_2026-09-01T162004.md)
- [results.json](https://synthcore.org/code/939/results.json)
- [Download all files as ZIP](https://synthcore.org/download/939)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.