Adaptive Proximal Quasi-Newton Training

1. Integration point: use this optimizer on one or more sensitive parameter blocks of a small MLP, transformer feed-forward layer, or low-rank adapter. Set $R(w)=\lambda\|w\|_1$ for a sparse adapter, use an indicator function for box-constrained weights, or set $R(w)=0$ for a pure quasi-Newton comparison. Do not form $H_t$ densely; begin with diagonal scaling, limited-memory BFGS pairs, or a compact two-loop inverse-Hessian application. 2. Pseudocode: compute $g=\operatorname{backward}(F,w)$; propose $z=w-\eta H g$; compute $w^{trial}=\operatorname{prox}^{H}_{\eta R}(z)$; evaluate a fresh or cached minibatch merit $M$; accept if $M(w^{trial})\leq M(w)+c\,g^T(w^{trial}-w)$ or if a fixed-point residual decreases; on acceptance update $(s,y)$ and increase the success counter; on rejection restore $w$, shrink $\eta$, and clear unstable curvature pairs. Enlarge the stepsize only after $q=3$ successes. 3. Computed versus estimated: gradients and objective values are computed by autodiff; curvature is estimated from successive gradients; $\lambda_{\max}$ is estimated with 5--10 Hessian-vector power iterations, not explicitly computed. 4. First cheap experiment: train a two-layer ReLU MLP on MNIST or CIFAR-10 using SGD, AdamW, fixed-step proximal gradient, and this method with identical minibatches. Sweep the initial $\eta$ and record accepted/rejected trials, loss, and gradient norm. The prediction is a sharp divergence or oscillation boundary near $\eta_c=2/\lambda_{\max}(H^{1/2}\nabla^2F H^{1/2})$ on a fixed small-batch quadratic or nearly quadratic network; the measured largest stable stepsize should agree with the power-iteration estimate within approximately 20%, while adaptive enlargement should settle within 10--20% below that boundary.