{
 "artifacts": null,
 "category": "optimization",
 "description": "Use two points with approximately equal minibatch loss to construct an ellipcenter: the intersection of the normal lines through the two points, where the normals are their gradients. The resulting update uses local curvature information in the span of two gradients and can be relaxed toward the current parameters or combined with momentum.",
 "formulas_latex": [
  "$$y_k=x_k-t_k\\nabla f(x_k),\\qquad f(y_k)=f(x_k)$$",
  "$$\\Pi_k:=\\left\\{x\\in\\mathbb{R}^n:x=x_k+\\operatorname{span}\\{\\nabla f(x_k),\\nabla f(y_k)\\}\\right\\}$$",
  "$$c_k=x_k+\\alpha_k g_k=y_k+\\beta_k g'_k,\\qquad g_k=\\nabla f(x_k),\\quad g'_k=\\nabla f(y_k)$$",
  "$$\\alpha_k=\\frac{(d_k^Tg_k)(g'_k{}^Tg'_k)-(d_k^Tg'_k)(g_k^Tg'_k)}{(g_k^Tg_k)(g'_k{}^Tg'_k)-(g_k^Tg'_k)^2},\\qquad d_k=y_k-x_k$$"
 ],
 "id": 2829,
 "implementation": "(1) Integration point: replace the parameter update of SGD or AdamW for a selected parameter block, preferably the final projection layers or all weights in a small-model MVP. The forward pass at $\\theta$ produces minibatch loss $L(\\theta)$ and gradient $g=\\nabla_\\theta L(\\theta)$. Create a trial point $\\theta'=\\theta-tg$ and evaluate the same minibatch loss and gradient there. (2) Pseudocode: `g = grad(L(theta)); t = t_init; repeat theta_trial = theta - t*g; loss_trial = L(theta_trial); if abs(loss_trial-loss) \u003c= tol*max(1,abs(loss)): break; if loss_trial \u003e loss: t *= 0.5; else: t *= 1.1; g2 = grad(L(theta_trial)); d = theta_trial-theta; A = dot(g,g); B = dot(g,g2); C = dot(g2,g2); D = A*C-B*B; if D \u003c eps*A*C: target = 0.5*(theta+theta_trial); else alpha=(dot(d,g)*C-dot(d,g2)*B)/D; target=theta+alpha*g; direction=target-theta; velocity=mu*velocity+rho*direction; theta += velocity`. (3) The paper's mathematics supplies the equal-level construction, affine gradient plane, normal-line intersection, and Gram determinant $D$. The line-search tolerance, initial step $t_init$, relaxation $\\rho$, momentum $\\mu$, and determinant threshold are empirical hyperparameters. Estimate numerical degeneracy using $D/(AC)$ and fall back to the midpoint when gradients are nearly collinear. For noisy minibatches, keep the same batch for both evaluations and optionally use an exponential moving average of the two losses. (4) First experiment: train a 2-layer MLP and a small ResNet on MNIST or CIFAR-10 using identical minibatch order, comparing SGD with momentum, AdamW, and this optimizer at equal gradient-evaluation budget. Try $\\rho\\in\\{0.25,0.5,1\\}$, $\\mu\\in\\{0,0.9\\}$, and $t_init$ equal to the current SGD learning rate. Success means faster training-loss decrease or higher validation accuracy at equal wall-clock time; failure signals are excessive second-forward cost, frequent degenerate Gram systems, or instability caused by noisy equal-loss line search.",
 "math_summary": "The paper's ME construction starts from a current point $x_k$ and a nonzero gradient $g_k=\\nabla f(x_k)$. It selects the unique point $y_k\\neq x_k$ on the negative-gradient ray satisfying $y_k=x_k-t_k g_k$ and $f(y_k)=f(x_k)$. If $g_k$ and $g'_k=\\nabla f(y_k)$ are linearly independent, the ellipse $E_k$ lies in the affine plane $\\Pi_k=\\{x:x=x_k+\\operatorname{span}\\{g_k,g'_k\\}\\}$ and is orthogonal to $g_k$ at $x_k$ and to $g'_k$ at $y_k$. For an ellipse, orthogonality means that its center $c_k$ lies on both normal lines, so $c_k=x_k+\\alpha_k g_k=y_k+\\beta_k g'_k$. Let $d_k=y_k-x_k$, $a=g_k^Tg_k$, $b=g_k^Tg'_k$, and $e=g'_k{}^Tg'_k$. Solving the two normal-line equations gives $\\alpha_k=((d_k^Tg_k)e-(d_k^Tg'_k)b)/(ae-b^2)$, provided the Gram determinant $ae-b^2\u003e0$. For a quadratic $f(x)=\\tfrac12x^TQx+q^Tx+r$ with positive-definite $Q$, the exact equal-level step is $t_k=2\\|g_k\\|^2/(g_k^TQg_k)$; in a neural network, replace this unavailable curvature expression by backtracking or a secant estimate. The relaxed momentum update is $\\theta_{k+1}=\\theta_k+\\rho_k(c_k-\\theta_k)+\\mu v_k$, with $0\u003c\\rho_k\\le1$ and $0\\le\\mu\u003c1$.",
 "math_tags": [
  "optimization",
  "geometry",
  "numerical-analysis",
  "linear-algebra"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.29454",
  "arxiv_url": "https://arxiv.org/abs/2608.29454",
  "summary_what_math_gives_to_ml": "The paper introduces an optimizer geometry that uses two points on the same objective level set and the intersection of their normal directions as an ellipcenter. For a strictly convex quadratic, the equal-value second point can be obtained exactly along the negative-gradient ray, and the two endpoint gradients define a two-dimensional subspace that captures local anisotropy. This suggests a minibatch optimizer that replaces part of a conventional gradient step with a curvature-aware secant construction, optionally mixed with momentum and relaxation. The main transferable asset is not the ellipse itself but the cheap use of two equal-loss gradient evaluations to infer a local two-dimensional geometry without forming a Hessian.",
  "title": "The method of ellipcenters with momentum and relaxation for convex quadratic minimization",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 7,
  "usefulness": 6
 },
 "solves": [
  "speedup",
  "stability",
  "accuracy"
 ],
 "title": "Ellipcenter Secant Optimizer",
 "url": "https://synthcore.org/idea/2829/ellipcenter-secant-optimizer",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}