Rank-One Proximal Quasi-Newton Optimizer

Integrate this into a proximal-gradient optimizer for a neural network with parameter vector $\theta$. Use the smooth loss $f(\theta)$, regularizer $h(\theta)$, diagonal metric $D=\operatorname{diag}(d)$, rank-one direction $a$, and nonnegative rank coefficient $\rho$. At iteration $t$, compute $g=\nabla f(\theta_t)$ and form the preconditioned gradient point $z=\theta_t-M^{-1}g$, using the Sherman-Morrison formula above. Then solve the proximal update: (1) initialize $q$ from the previous iteration's root; (2) evaluate $x(q)=P_D(z-D^{-1}a q)$ with the existing diagonal proximal kernel; (3) evaluate $r(q)=q-\rho a^\top(x(q)-z)$; (4) bracket a sign change by expanding an interval around the warm start; (5) run 5--20 bisection or safeguarded secant steps; and (6) set $\theta_{t+1}=x(q)$. For group lasso, implement the diagonal proximal map blockwise; for an arbitrary structured penalty, call its ordinary diagonal-metric proximal routine. Estimate $d_i$ from Adam's second-moment accumulator or RMSProp, clipped to $[d_{\min},d_{\max}]$. Obtain $a$ from recent curvature, for example $s=\theta_t-\theta_{t-1}$ and $y=g_t-g_{t-1}$ with $a=y/(\|y\|_2+\varepsilon)$; choose $\rho=\max(0,(s^\top y)/(s^\top s+\varepsilon)-\bar d)$ and clip it to preserve conditioning. The exact mathematical components are the residual solve and Sherman-Morrison inverse; the diagonal estimator, clipping, and rank-one refresh frequency are empirical. First test a 2-layer MLP on MNIST or CIFAR-10 with group-lasso or synthetic block sparsity, comparing diagonal proximal gradient, AdamW, and an L-BFGS-style baseline at equal forward/backward FLOPs. Record loss decrease per step, wall-clock time, validation accuracy, number of proximal evaluations, and residual $|r(q)|$. Success means faster loss descent or stronger structured sparsity at equal compute without divergence; target 10--20% fewer iterations than the diagonal method with less than 10% proximal overhead.