# Conjugate-Free Secant Preconditioner

- ID: 2036
- Canonical URL: https://synthcore.org/idea/2036/conjugate-free-secant-preconditioner
- API JSON: https://synthcore.org/api/idea/2036.json
- API Markdown: https://synthcore.org/api/idea/2036.md
- Verification status: unverified
- Source: [arXiv:2608.09206](https://arxiv.org/abs/2608.09206)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, training-dynamics, mlp
- Math tags: optimization, convex-analysis, linear-algebra, numerical-analysis
- Ratings: usefulness 6/10; difficulty 6/10; novelty 5/10

## Idea description

Replace the purely diagonal preconditioner in AdamW or SGD with a blockwise, single-secant BFGS inverse-curvature metric. Use spectral damping and clipping relative to the diagonal RMS metric so the learned metric cannot become arbitrarily ill-conditioned, mirroring the paper's uniform comparison between its conjugate-free scaling and the primal barrier Hessian.

## Mathematical statement

The paper solves the primal-dual conic pair $\min\{\langle c,x\rangle:Ax=b,\ x\in\mathbb K\}$ and $\max\{\langle b,y\rangle:A^\top y+s=c,\ s\in\mathbb K^*\}$ using a scaling matrix obtained from a single-secant BFGS update of a primal barrier Hessian, avoiding conjugate-barrier derivatives. The transferable object is the inverse-BFGS secant map: for parameter displacement $u_t=\theta_t-\theta_{t-1}$ and gradient displacement $v_t=g_t-g_{t-1}$, with $g_t=\nabla_\theta L(\theta_t)$ and $\rho_t=(v_t^\top u_t)^{-1}$, $H_{t+1}=(I-\rho_tu_tv_t^\top)H_t(I-\rho_tv_tu_t^\top)+\rho_tu_tu_t^\top$. If $v_t^\top u_t>0$, this is positive definite and satisfies $H_{t+1}v_t=u_t$. In the neural adaptation, $D_t$ is a positive diagonal RMS/Adam metric and $H_t$ is a blockwise low-rank correction. Enforce the analogue of uniform comparability by damping and clipping eigenvalues so that $\alpha D_t^{-1}\preceq H_t\preceq\beta D_t^{-1}$ for fixed $0<\alpha\leq\beta$; this bounds the update direction and prevents unstable curvature amplification.

## Key formulas

- $$\min\{\langle c,x\rangle:Ax=b,\ x\in\mathbb K\},\qquad \max\{\langle b,y\rangle:A^\top y+s=c,\ s\in\mathbb K^*\}.$$
- $$H_{t+1}=(I-\rho_tu_tv_t^\top)H_t(I-\rho_tv_tu_t^\top)+\rho_tu_tu_t^\top,\qquad \rho_t=(v_t^\top u_t)^{-1},\quad H_{t+1}v_t=u_t.$$
- $$\widetilde H_t=D_t^{-1/2}H_tD_t^{-1/2},\qquad \widetilde H_t\leftarrow Q\,\operatorname{diag}(\operatorname{clip}(\lambda_i,\alpha,\beta))Q^\top,$$
- $$\theta_{t+1}=\theta_t-\eta_t H_t g_t,\qquad \alpha D_t^{-1}\preceq H_t\preceq\beta D_t^{-1}.$$

## Implementation notes

Integrate this into the optimizer update for each trainable layer, initially on a small MLP or Transformer feed-forward block rather than on the entire parameter vector. Maintain Adam's exponential second moment $r_t=\beta_2r_{t-1}+(1-\beta_2)g_t^2$ and set $D_t=\operatorname{diag}(\sqrt{r_t}+\epsilon)$. Flatten each selected layer into blocks of 1,024--4,096 parameters. Store one dense symmetric inverse metric $H_{t,b}$ per block, initialized to $D_{t,b}^{-1}$, or use a diagonal matrix plus a rank-two correction. At step $t$, save the previous block parameters and gradients, compute $u_t=\theta_t-\theta_{t-1}$ and $v_t=g_t-g_{t-1}$, and skip the update if $v_t^\top u_t\leq\delta\|u_t\|\|v_t\|$ for small $\delta$, because the secant curvature is nonpositive or numerically unreliable. Otherwise apply inverse BFGS with $\rho_t=1/(v_t^\top u_t)$. Every 10--100 steps, whiten the block metric as $\widetilde H=D^{1/2}HD^{1/2}$, estimate its extremal eigenvalues using 3--5 power iterations, and clip them to $[\alpha,\beta]$, for example $[0.1,10]$. Use $H_tg_t$ instead of $D_t^{-1}g_t$ in the AdamW update while retaining decoupled weight decay. The paper supplies the secant construction and metric-comparability rationale; eigenvalues and skipped-update rates are estimated empirically. First test on CIFAR-10 with a 2-layer MLP and a 6-layer small Transformer, comparing AdamW, Shampoo, and diagonal Adam at equal wall-clock and FLOPs. Success means faster loss decrease at equal optimizer steps without gradient explosions, together with improved final accuracy; failure is overhead dominating or frequent curvature skips.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.