{
 "artifacts": null,
 "category": "optimization",
 "description": "Replace a full Hermitian curvature matrix, such as a Hessian or empirical Fisher matrix, by its block-diagonal version only when the paper's perturbation certificate predicts a small eigenvalue change. Use the certificate online to merge poorly separated blocks and retain independent preconditioners for well-separated blocks, yielding a controllable accuracy-memory tradeoff rather than a fixed block-diagonal approximation.",
 "formulas_latex": [
  "$$A=\\begin{bmatrix}H_{1}\u0026E^{*}\\\\E\u0026H_{2}\\end{bmatrix},\\qquad \\widetilde{A}=\\begin{bmatrix}H_{1}\u00260\\\\0\u0026H_{2}\\end{bmatrix},\\qquad V=A-\\widetilde{A}.$$",
  "$$\\phi(\\delta,\\epsilon)=\\frac{2\\epsilon}{\\delta+\\sqrt{\\delta^{2}+4\\epsilon^{2}}},\\qquad \\eta=\\min_{\\lambda\\in\\eig(H_1),\\,\\mu\\in\\eig(H_2)}|\\lambda-\\mu|,\\qquad \\epsilon_2=\\|E\\|_2.$$",
  "$$\\left\\|\\operatorname{diag}\\!\\left(\\boldsymbol{\\lambda}(A)-\\boldsymbol{\\lambda}(\\widetilde{A})\\right)\\right\\|_{\\mathrm{UI}}\\leq \\phi(\\eta,\\epsilon_2)\\,\\|A-\\widetilde{A}\\|_{\\mathrm{UI}}.$$",
  "$$\\widehat{\\Delta}_{\\mathrm{curv}}=\\phi(\\widehat\\eta,\\widehat\\epsilon_2)\\,\\|\\widehat V\\|_{F},\\qquad \\text{accept block deletion only if }\\widehat{\\Delta}_{\\mathrm{curv}}\\leq\\tau.$$"
 ],
 "id": 2802,
 "implementation": "Integrate this into a second-order optimizer or curvature-aware Adam variant at the preconditioner-construction step. Partition parameters into blocks, such as one block per layer or groups of 1–10 million parameters, and maintain a block-diagonal approximation to a damped empirical Fisher matrix \\(A=G+\\lambda I\\), where \\(G\\) is estimated from minibatch gradients and \\(\\lambda\u003e0\\) ensures positive definiteness. For each candidate pair of blocks, estimate the within-block matrices \\(H_1,H_2\\) and cross-block matrix \\(E\\) using gradient-covariance sketches instead of materializing the full matrix. Estimate \\(\\widehat\\epsilon_2=\\|E\\|_2\\) by randomized power iteration: initialize a random unit vector \\(v\\), repeatedly compute \\(v\\leftarrow E^*Ev/\\|E^*Ev\\|_2\\), and return \\(\\widehat\\epsilon_2=\\|Ev\\|_2\\). Estimate extremal eigenvalues of each \\(H_i\\) with a few Lanczos iterations; set \\(\\widehat\\eta\\) to the smallest pairwise distance between these estimates, or conservatively set it to zero if spectral intervals overlap. Compute \\(\\phi(\\widehat\\eta,\\widehat\\epsilon_2)\\) and \\(\\widehat\\Delta_{\\mathrm{curv}}=\\phi(\\widehat\\eta,\\widehat\\epsilon_2)\\|\\widehat V\\|_F\\). Pseudocode: collect gradient sketches; estimate \\(H_i\\), \\(E\\), \\(\\widehat\\eta\\), and \\(\\widehat\\epsilon_2\\); if the certificate is below threshold \\(\\tau\\), retain independent block inverses or Kronecker factors; otherwise merge the pair and construct a joint factor; apply the resulting preconditioner to the gradient. The theorem supplies the perturbation bound, while all spectral quantities are estimated empirically. First test on a six-layer MLP or small ViT on CIFAR-10, comparing full-matrix, fixed layerwise block-diagonal, and adaptive certified preconditioning at equal optimizer memory. Measure loss versus optimizer FLOPs, final accuracy, estimated curvature-eigenvalue distortion, and peak memory. A successful result is at least 2x lower preconditioner memory with no more than 1% relative loss in final accuracy, while the certificate correctly predicts block merges preceding optimization instability.",
 "math_summary": "Let \\(A=\\begin{bmatrix}H_1\u0026E^*\\\\E\u0026H_2\\end{bmatrix}\\) be a Hermitian curvature matrix, where \\(H_1\\) and \\(H_2\\) are within-block curvature matrices and \\(E\\) contains cross-block curvature. Let \\(\\widetilde A=\\begin{bmatrix}H_1\u00260\\\\0\u0026H_2\\end{bmatrix}\\), \\(V=A-\\widetilde A\\), and \\(\\epsilon_2=\\|E\\|_2=\\|V\\|_2\\). Define the spectral gap \\(\\eta=\\min_{\\lambda\\in\\eig(H_1),\\,\\mu\\in\\eig(H_2)}|\\lambda-\\mu|\\). The paper defines \\(\\phi(\\delta,\\epsilon)=\\frac{2\\epsilon}{\\delta+\\sqrt{\\delta^2+4\\epsilon^2}}\\) for \\((\\delta,\\epsilon)\\neq(0,0)\\), and \\(\\phi(0,0)=1\\). For every Q-norm, including Schatten \\(p\\)-norms for \\(2\\leq p\\leq\\infty\\), the sharp bound is \\(\\|\\operatorname{diag}(\\boldsymbol\\lambda(A)-\\boldsymbol\\lambda(\\widetilde A))\\|_{\\mathrm{UI}}\\leq\\phi(\\eta,\\epsilon_2)\\|A-\\widetilde A\\|_{\\mathrm{UI}}\\); it also holds for every unitarily invariant norm when \\(\\operatorname{rank}(E)\\leq1\\). Here \\(\\boldsymbol\\lambda(\\cdot)\\) is the descending eigenvalue vector and \\(\\operatorname{diag}(x)\\) is the diagonal matrix with entries \\(x\\). Use the Frobenius or spectral norm to obtain a computable upper bound on curvature-spectrum distortion.",
 "math_tags": [
  "spectral-theory",
  "linear-algebra",
  "approximation-theory",
  "optimization"
 ],
 "ml_areas": [
  "optimizer",
  "memory",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.29009",
  "arxiv_url": "https://arxiv.org/abs/2608.29009",
  "summary_what_math_gives_to_ml": "The paper gives a sharp, dimension-independent certificate for deleting off-diagonal blocks from a Hermitian matrix: the induced eigenvalue displacement is controlled by both the cross-block norm and the spectral separation between diagonal blocks. This is directly useful for block-diagonal approximations of Hessians, Fisher matrices, Gram matrices, and covariance-based neural optimizers, where expensive curvature objects are replaced by independent parameter-block statistics. The transferable asset is the explicit factor \\(\\phi(\\eta,\\epsilon)\\), which becomes small when blocks are spectrally separated and prevents unjustified block decoupling when they are not. The best initial application is an adaptive block-diagonal preconditioner that merges parameter blocks when the certificate predicts unacceptable curvature-spectrum distortion.",
  "title": "A Sharp Unitarily Invariant Norm Bound for the Off-Diagonal Block Perturbation of a Hermitian Matrix",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 5,
  "usefulness": 7
 },
 "solves": [
  "memory",
  "speedup",
  "stability"
 ],
 "title": "Spectral-Certified Block-Diagonal Preconditioning",
 "url": "https://synthcore.org/idea/2802/spectral-certified-block-diagonal-preconditioning",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}