Conjugate-Free Secant Preconditioner

Integrate this into the optimizer update for each trainable layer, initially on a small MLP or Transformer feed-forward block rather than on the entire parameter vector. Maintain Adam's exponential second moment $r_t=\beta_2r_{t-1}+(1-\beta_2)g_t^2$ and set $D_t=\operatorname{diag}(\sqrt{r_t}+\epsilon)$. Flatten each selected layer into blocks of 1,024--4,096 parameters. Store one dense symmetric inverse metric $H_{t,b}$ per block, initialized to $D_{t,b}^{-1}$, or use a diagonal matrix plus a rank-two correction. At step $t$, save the previous block parameters and gradients, compute $u_t=\theta_t-\theta_{t-1}$ and $v_t=g_t-g_{t-1}$, and skip the update if $v_t^\top u_t\leq\delta\|u_t\|\|v_t\|$ for small $\delta$, because the secant curvature is nonpositive or numerically unreliable. Otherwise apply inverse BFGS with $\rho_t=1/(v_t^\top u_t)$. Every 10--100 steps, whiten the block metric as $\widetilde H=D^{1/2}HD^{1/2}$, estimate its extremal eigenvalues using 3--5 power iterations, and clip them to $[\alpha,\beta]$, for example $[0.1,10]$. Use $H_tg_t$ instead of $D_t^{-1}g_t$ in the AdamW update while retaining decoupled weight decay. The paper supplies the secant construction and metric-comparability rationale; eigenvalues and skipped-update rates are estimated empirically. First test on CIFAR-10 with a 2-layer MLP and a 6-layer small Transformer, comparing AdamW, Shampoo, and diagonal Adam at equal wall-clock and FLOPs. Success means faster loss decrease at equal optimizer steps without gradient explosions, together with improved final accuracy; failure is overhead dominating or frequent curvature skips.