✗ Mechanism failed 2026

Curvature-Band SAM Direction

Implementation & benchmark of arXiv:2608.24444 — Spectral Selection in Sphere-Constrained Flows Generated by Polynomials of the Dirichlet Laplacian

Usefulness6/10

Difficulty6/10

Novelty7/10

Regularization Stability Generalization Optimizer Regularization Training dynamics Spectral theory Linear algebra Optimization Dynamical systems

Source paper: Spectral Selection in Sphere-Constrained Flows Generated by Polynomials of the Dirichlet Laplacian arXiv:2608.24444 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the random or gradient-aligned perturbation in sharpness-aware minimization with a unit perturbation direction selected by a polynomial of the local Hessian. With $\mathscr{P}(s)=(s-\rho)^2$, the direction converges toward Hessian eigenspaces whose eigenvalues are closest to the target curvature $\rho$, allowing regularization of a chosen curvature band instead of indiscriminately penalizing only the sharpest direction.

Formulas

$$u_t=-\Pi_u\mathscr{P}(\mathcal{B})u,\qquad \Pi_uv=v-\langle v,u\rangle_Hu,$$

$$u(t)=\frac{e^{-t\mathscr{P}(\mathcal{B})}u_0}{\left\|e^{-t\mathscr{P}(\mathcal{B})}u_0\right\|_H},$$

$$\mathscr{P}(s)=(s-\rho)^2,\qquad \mathscr{P}(H)u=H^2u-2\rho Hu+\rho^2u,$$

$$u_{k+1}=\frac{u_k-\eta\left(H^2u_k-2\rho Hu_k+\rho^2u_k\right)}{\left\|u_k-\eta\left(H^2u_k-2\rho Hu_k+\rho^2u_k\right)\right\|_2}.$$

Mathematical statement

The paper defines a self-adjoint operator $\mathcal{B}e_j=\lambda_j e_j$, a polynomial $\mathscr{P}(s)=a_ms^m+\cdots+a_0$ with $a_m>0$, and the sphere-constrained flow $u_t=-\Pi_u\mathscr{P}(\mathcal{B})u$, where $\Pi_uv=v-\langle v,u\rangle_Hu$. Its exact solution is $u(t)=e^{-t\mathscr{P}(\mathcal{B})}u_0/\|e^{-t\mathscr{P}(\mathcal{B})}u_0\|_H$. If $u_0=\sum_j c_je_j$, every initially active eigenspace remains active and the normalized trajectory converges to the span of active eigenvectors minimizing $\mathscr{P}(\lambda_j)$. For $\mathscr{P}(s)=(s-\rho)^2$, selected eigenvalues minimize squared distance to $\rho$; if the minimum is unique and the next active polynomial value differs by $\Delta>0$, unwanted components decay exponentially at rate $\Delta$. The neural adaptation sets $\mathcal{B}=H$, the local Hessian, and computes $H^2u$ using two Hessian-vector products.

Implementation notes

Integrate this at the perturbation-direction line of SAM, before the second forward/backward pass. Let $\theta$ be model parameters, $L(\theta)$ the minibatch loss, $g=\nabla_\theta L$, and $u$ a persistent unit vector shaped like $\theta$. Every $T$ optimizer steps, initialize $u_0=g/(\|g\|_2+\epsilon)$, or reuse the previous unit direction. Run $K=2$ to $5$ iterations of the displayed update. Each iteration computes $v=Hu$ with one Hessian-vector product and $w=Hv$ with a second Hessian-vector product, forms $q=w-2\rho v+\rho^2u$, updates $u\leftarrow u-\eta q$, and renormalizes. Stop gradients through the Hessian construction and keep norm reductions in FP32. Estimate $\rho$ from several random HVP probes or from the running median of Rayleigh quotients $r_i=v_i^THv_i$ for unit probes $v_i$. Form the SAM perturbation $\delta=\gamma u$, evaluate $L(\theta+\delta)$, and apply the ordinary gradient update from the perturbed loss. The paper supplies the spectral selection mechanism and exponential-gap intuition; HVPs and $\rho$ are estimated numerically. First test a 2-layer MLP or small ViT on CIFAR-10, comparing SGD, SAM, and curvature-band SAM at equal optimizer steps and matched HVP budgets. Measure training loss, test accuracy, top Hessian eigenvalue, loss under random perturbations, and the Rayleigh quotient of $u$. Success means equal-or-better accuracy with lower directional sharpness or more stable loss curves than SAM.

Verification

Mechanism failed

Stage 1 · Toy mechanism gate: Failed ✗

Stage 2 · Mechanism transferred to benchmark: Not tested

Stage 2 · Practical benchmark result: Not run

Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a Hessian-polynomial curvature-band SAM prototype with two HVPs per polynomial iteration, plus an exact finite-dimensional spectral sanity check. The math claim was clearly observed: for rho=3, eigenvalues 2 and 4 were selected and the unwanted-component ratio fell from 1.19 to 1.77e-11, consistent with the polynomial gap. However, on the tiny two-moons MLP, curvature-band SAM matched SAM/SGD accuracy (0.844) but had higher random-perturbation loss increase (1.63e-4 vs SAM 1.18e-4), so there is no observed practical win.

Agent confidence: 9/10

Baseline: SGD: test loss 0.3561, accuracy 0.8438, random perturbation loss increase 1.20e-4
Idea: Curvature-band SAM: test loss 0.3549, accuracy 0.8438, random perturbation loss increase 1.63e-4, mean selected-direction Rayleigh quotient 0.419; spectral exact ratio 1.19 -> 1.77e-11

Limitations: Only one tiny two-moons dataset, one seed, and a small MLP were tested; CIFAR-10, multiple seeds, top-Hessian evaluation, persistent directions, and systematic rho/eta/K/gamma tuning were not tested. The rho estimate used only two random HVP probes, and the reported curvature comparison uses a small random-perturbation proxy.

How to run: python3 experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Artifacts

Implementation overview ⬇ Download all as ZIP 3 files · code, reports and structured results

🐍 experiment.py 6.4 KB View Raw
📄 report.md 1.4 KB View
📊 results.json 1.2 KB View Raw