{
 "artifacts": [
  {
   "name": "experiment.py",
   "url": "https://synthcore.org/code/16/experiment.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/16/report.md"
  }
 ],
 "category": "optimization",
 "description": "Use one projected LM direction and let Armijo backtracking decide the accepted step length, instead of repeatedly resolving the damped system whenever a trial step is rejected. Independently update the damping parameter from the ratio of actual to predicted reduction, preserving the trust-region-like adaptivity of LM while reducing expensive curvature solves.",
 "download_zip": "https://synthcore.org/download/16",
 "formulas_latex": [
  "$$m(p)=F(\\theta)+g^Tp+\\frac{1}{2}p^THp,\\qquad \\operatorname{Pred}=-g^Tp-\\frac{1}{2}p^THp.$$",
  "$$\\operatorname{Act}(\\alpha)=F(\\theta)-F(\\theta+\\alpha p),\\qquad \\rho(\\alpha)=\\frac{\\operatorname{Act}(\\alpha)}{\\operatorname{Pred}}.$$",
  "$$F(\\theta+\\alpha p)\\le F(\\theta)+c\\alpha g^Tp,\\qquad \\alpha\\in\\{1,\\beta,\\beta^2,\\ldots\\}.$$",
  "$$\\lambda_{t+1}=\\begin{cases}\\gamma_{\\mathrm{up}}\\lambda_t,\u0026\\rho_t\u003c\\rho_{\\mathrm{low}},\\\\ \\lambda_t/\\gamma_{\\mathrm{down}},\u0026\\rho_t\u003e\\rho_{\\mathrm{high}},\\\\ \\lambda_t,\u0026\\text{otherwise}.\\end{cases}$$"
 ],
 "id": 176,
 "implementation": "1. Integration point: modify the optimizer step after any subspace, diagonal, or Krylov LM direction has been computed. This can be used independently of the hybrid basis, but the intended integration is with the projected solve (U^THU+lambda I)z=-U^Tg. 2. Pseudocode: at iteration t, compute F(theta), g, and one damped direction p. Compute Pred=max(epsilon, -g dot p - 0.5 p dot H p), where Hp is obtained either from the already available projected curvature or one additional JVP/VJP. Set alpha=1. While F(theta+alpha*p)\u003eF(theta)+c*alpha*(g dot p) and alpha\u003ealpha_min, replace alpha by beta*alpha. If no acceptable alpha exists, skip the update and increase lambda. Otherwise set theta_new=theta+alpha*p, compute Act=F(theta)-F(theta_new), rho=Act/Pred, and update lambda using rho_low, rho_high, gamma_up, and gamma_down. Do not recompute p during backtracking. 3. Computed from the math: Armijo acceptance, the model-prediction term, and rho. Estimated empirically: F on each trial minibatch; to reduce noise, use the same minibatch for all trial evaluations or maintain an exponential moving average of Act and Pred. Use c=1e-4, beta=0.5, alpha_min=1/32, rho_low=0.25, rho_high=0.75, gamma_up=4, and gamma_down=2. 4. First cheap experiment: compare this policy against conventional LM that changes lambda and resolves the projected system after every rejected trial, using the same 2-layer MLP and regression dataset. Record wall-clock time, number of curvature products, rejected trials, accepted loss decrease, and final MSE. The expected signal is similar or better loss per accepted iteration with substantially fewer projected solves, especially when minibatch curvature makes full-step acceptance unreliable.",
 "math_summary": "Let F(theta)=1/2||r(theta)||_2^2, g=∇F(theta), H=J^TJ, and p be a direction obtained from one projected LM solve. The quadratic model is m(p)=F(theta)+g^Tp+1/2 p^THp, so the predicted reduction is Pred=-g^Tp-1/2 p^THp. For a trial step alpha p, the actual reduction is Act(alpha)=F(theta)-F(theta+alpha p), and the reduction ratio is rho(alpha)=Act(alpha)/Pred. Armijo acceptance chooses the largest alpha from {1,beta,beta^2,...}, with 0\u003cbeta\u003c1, satisfying F(theta+alpha p)\u003c=F(theta)+c alpha g^Tp for 0\u003cc\u003c1/2. Crucially, alpha is selected without changing lambda or recomputing p. After acceptance, update lambda only from rho, for example lambda\u003c-gamma_up lambda when rho\u003crho_low, lambda\u003c-lambda/gamma_down when rho\u003erho_high, and leave it unchanged otherwise. The mathematical benefit is that line-search globalization and curvature regularization are separate control loops: rejected lengths do not trigger repeated linear solves.",
 "math_tags": [
  "optimization",
  "numerical-analysis",
  "dynamical-systems"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics",
  "mlp"
 ],
 "paper": {
  "arxiv_id": "2608.25524",
  "arxiv_url": "https://arxiv.org/abs/2608.25524",
  "summary_what_math_gives_to_ml": "The paper offers a practical large-scale second-order optimizer built around an adaptively enriched low-dimensional parameter subspace rather than a full Gauss–Newton solve. Its transferable asset is the explicit combination of descent directions, accepted-step memory, Krylov curvature probes, and randomized curvature directions, together with a projected-gradient adequacy test that detects when the subspace is missing important optimization information. A second useful mechanism separates step acceptance from damping adaptation: Armijo backtracking chooses the step length, while the actual-to-predicted reduction ratio changes the LM damping without repeatedly solving new damped systems. The most direct neural-network test is a minibatch HSLM optimizer on small and medium regression models, measuring loss decrease per wall-clock time and per Jacobian-vector product against AdamW, full LM, and Krylov LM.",
  "title": "Adaptive Hybrid Subspace Levenberg Marquardt Algorithm with Adequacy Monitor for Large Scale Least Squares Problems",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 8,
  "usefulness": 7
 },
 "solves": [
  "speedup",
  "stability"
 ],
 "title": "Armijo acceptance with decoupled LM damping",
 "url": "https://synthcore.org/idea/176/armijo-acceptance-with-decoupled-lm-damping",
 "verification": {
  "peer_reviewed": false,
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 8,
   "verdict": "Built a NumPy nonlinear least-squares benchmark comparing decoupled Armijo LM against conventional LM that increases damping and re-solves after rejected full steps. The quadratic-model and reduction-ratio identities passed at approximately 1e-15 error. Across five seeds, the idea used 35 solves every run versus 42.6 baseline solves on average (17.6% fewer), and mean final loss was lower (0.1181 versus 0.1426), but it won on final loss only 2/5 runs and was generally not faster because Armijo backtracking added function evaluations.",
   "metrics": {
    "baseline": "5-seed mean final loss 0.1426035; mean 42.6 LM solves; mean rejected trials 7.6; representative runtime 1.067 s",
    "idea": "5-seed mean final loss 0.1181096; exactly 35 LM solves; mean rejected trials 12.6; representative runtime 0.977 s; lower loss in 2/5 seeds"
   },
   "how_to_run": "python3 experiment.py",
   "files": [
    "experiment.py"
   ],
   "limitations": "Tiny full-batch NumPy MLP regression only; identity projection was used rather than a reduced subspace; no minibatch noise, JVP/VJP implementation, FLOP-normalized benchmark, or larger neural network was tested. Runtime measurements are noisy and the baseline/idea use different numbers of objective evaluations."
  },
  "status": "mechanism_works",
  "status_label": "Mechanism works",
  "updated_at": "2026-08-29T20:02:09",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": true,
    "tested": true
   }
  }
 }
}