{
 "artifacts": null,
 "category": "optimization",
 "description": "Use eigenvector deflation inside damped Newton or Gauss-Newton training to remove low-curvature directions that dominate the condition number of the inner linear solve. The optimizer solves the remaining curvature system with CG and explicitly reconstructs the contribution from the deflated subspace, reducing the number of Hessian-vector products needed for a useful second-order step.",
 "formulas_latex": [
  "$$Q=USV^{T}=\\sum_{k=1}^{\\min(m,n)}{\\bf u}_{k}\\sigma_{k}{\\bf v}_{k}^{T}.$$",
  "$$\\left\\|Q-\\sum_{k=1}^{r}{\\bf u}_{k}\\sigma_{k}{\\bf v}_{k}^{T}\\right\\|^{2}_{F}=\\left\\|\\sum_{k=r+1}^{\\min(m,n)}{\\bf u}_{k}\\sigma_{k}{\\bf v}_{k}^{T}\\right\\|^{2}_{F}\u003c\\tau_{\\sigma}^{2}.$$",
  "$$\\operatorname{compression\\ ratio}=\\frac{r(n+m+1)}{nm}.$$",
  "$$A=H+\\lambda I,\\qquad E=Z^{T}AZ,\\qquad s=ZE^{-1}Z^{T}b+\\left(I-ZE^{-1}Z^{T}A\\right)y,\\quad Ay\\approx b.$$"
 ],
 "id": 286,
 "implementation": "Integrate this method into a damped Newton, Gauss-Newton, or natural-gradient optimizer at the inner linear-solve stage, rather than changing the network forward pass. Given parameters theta, gradient g, damping lambda, and an inner tolerance, define the matrix-free operator A(v)=H(v)+lambda*v using automatic-differentiation Hessian-vector products or Gauss-Newton-vector products. Every T optimizer steps, run k-step Lanczos with these HVPs and retain the approximate eigenvectors Z corresponding to the smallest eigenvalues of A; reorthogonalize Z with QR. Apply A to every column of Z to form E=Z^T A Z and factor the small k-by-k matrix E with Cholesky. For each Newton right-hand side b=-g, compute the coarse component s0=Z E^{-1} Z^T b. Initialize CG with residual r0=b-A s0 and use the projected operator v -\u003e A(v)-A Z(E^{-1}(Z^T A(v))). Return s=s0+y, where y is the projected CG result. In finite precision, periodically reproject y against Z. Use a line search or trust-region acceptance test before updating theta. The mathematical quantities are the Lanczos subspace, coarse matrix, and projected operator; choose k, lambda, refresh period T, and CG tolerance empirically. For matrix-shaped parameters, optionally reshape each eigenvector into Q and retain SVD factors until the discarded Frobenius norm is below tau_sigma. Start with a two-layer MLP on MNIST or a small CIFAR-10 CNN. Compare ordinary CG, deflated CG with k in {4,8,16}, and Adam under identical HVP and wall-clock budgets. Record CG iterations, HVP count, loss versus time, final accuracy, and rejected line-search steps. Success is at least 25 percent fewer HVPs at matched loss or accuracy, without more rejected steps; also check whether the estimated post-deflation condition number lambda_max/lambda_{k+1} predicts the observed CG speedup.",
 "math_summary": "The central mechanism is direct eigenvector deflation: eigenvectors associated with problematic eigenvalues are projected out of the iterative solve while being retained as an explicit coarse correction space. For a neural-network step, let H be a symmetric Hessian, Gauss-Newton, or Fisher operator, let lambda be a positive damping parameter, and define A=H+lambda I, where I is the identity and the parameter dimension is p. Let Z=[z_1,...,z_k] in R^{p x k} contain approximate eigenvectors of A associated with its smallest eigenvalues, and define E=Z^T A Z. For right-hand side b=-g, where g is the gradient, the coarse correction is Z E^{-1} Z^T b. The complementary solve uses the projected operator P_A=I-Z E^{-1} Z^T A and returns s=Z E^{-1} Z^T b+P_A y, where y approximately solves A y=b. If Z accurately spans the first k eigendirections, the effective condition number changes from kappa(A)=lambda_max(A)/lambda_min(A) to approximately lambda_max(A)/lambda_{k+1}(A). The paper also provides SVD compression for a matrix-shaped eigenvector Q in R^{m x n}: Q=USV^T=sum_j u_j sigma_j v_j^T, with rank r selected so that the discarded Frobenius norm is below tau_sigma and storage ratio is r(n+m+1)/(nm).",
 "math_tags": [
  "linear-algebra",
  "spectral-theory",
  "numerical-analysis",
  "tensor-decomposition"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics",
  "fine-tuning"
 ],
 "paper": {
  "arxiv_id": "2606.31842",
  "arxiv_url": "https://arxiv.org/abs/2606.31842",
  "summary_what_math_gives_to_ml": "The paper develops a concrete spectral acceleration mechanism: explicitly remove troublesome eigenmodes from a slowly converging iteration, then solve the remaining problem in the complementary subspace. The transferable asset is the combination of targeted eigenvector deflation, augmented Krylov correction, and low-rank storage of deflation vectors. A promising neural-network application is damped Newton, Gauss-Newton, or natural-gradient training, where small-curvature Hessian or Fisher directions make inner conjugate-gradient solves poorly conditioned. The first test should use Hessian-vector products to estimate a few low-curvature modes and compare deflated versus ordinary CG at equal HVP and wall-clock budgets.",
  "title": "Numerical Study of Eigenvector Deflation to Accelerate the WaveHoltz Method",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 6,
  "usefulness": 6
 },
 "solves": [
  "speedup",
  "stability"
 ],
 "title": "Deflated Hessian-CG optimizer",
 "url": "https://synthcore.org/idea/286/deflated-hessian-cg-optimizer",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}