Ellipcenter Secant Optimizer

(1) Integration point: replace the parameter update of SGD or AdamW for a selected parameter block, preferably the final projection layers or all weights in a small-model MVP. The forward pass at $\theta$ produces minibatch loss $L(\theta)$ and gradient $g=\nabla_\theta L(\theta)$. Create a trial point $\theta'=\theta-tg$ and evaluate the same minibatch loss and gradient there. (2) Pseudocode: g = grad(L(theta)); t = t_init; repeat theta_trial = theta - t*g; loss_trial = L(theta_trial); if abs(loss_trial-loss) <= tol*max(1,abs(loss)): break; if loss_trial > loss: t *= 0.5; else: t *= 1.1; g2 = grad(L(theta_trial)); d = theta_trial-theta; A = dot(g,g); B = dot(g,g2); C = dot(g2,g2); D = A*C-B*B; if D < eps*A*C: target = 0.5*(theta+theta_trial); else alpha=(dot(d,g)*C-dot(d,g2)*B)/D; target=theta+alpha*g; direction=target-theta; velocity=mu*velocity+rho*direction; theta += velocity. (3) The paper's mathematics supplies the equal-level construction, affine gradient plane, normal-line intersection, and Gram determinant $D$. The line-search tolerance, initial step $t_init$, relaxation $\rho$, momentum $\mu$, and determinant threshold are empirical hyperparameters. Estimate numerical degeneracy using $D/(AC)$ and fall back to the midpoint when gradients are nearly collinear. For noisy minibatches, keep the same batch for both evaluations and optionally use an exponential moving average of the two losses. (4) First experiment: train a 2-layer MLP and a small ResNet on MNIST or CIFAR-10 using identical minibatch order, comparing SGD with momentum, AdamW, and this optimizer at equal gradient-evaluation budget. Try $\rho\in\{0.25,0.5,1\}$, $\mu\in\{0,0.9\}$, and $t_init$ equal to the current SGD learning rate. Success means faster training-loss decrease or higher validation accuracy at equal wall-clock time; failure signals are excessive second-forward cost, frequent degenerate Gram systems, or instability caused by noisy equal-loss line search.