Unverified 2026

Sketched curvature-subspace optimizer

Implementation & benchmark of arXiv:2608.24529 — Randomized Block Davidson Eigensolvers for Plane-Wave Density-Functional Theory

Usefulness5/10
Difficulty6/10
Novelty7/10

Source paper: Randomized Block Davidson Eigensolvers for Plane-Wave Density-Functional Theory arXiv:2608.24529 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Construct a block of gradient, preconditioned-gradient, or Hessian-vector-product directions without performing full-dimensional Gram-Schmidt. Use a random sketch to orthogonalize the block cheaply, then solve a small generalized eigenproblem using the true parameter-space overlap matrix so the extracted curvature modes are accurate for the generated subspace. Use the selected curvature modes to form a damped or trust-region optimizer step.

Formulas

$$\langle x,y\rangle_{\Omega}:=(\Omega x)^T(\Omega y),\qquad (1-\varepsilon)\|x\|_2^2\leq\|\Omega x\|_2^2\leq(1+\varepsilon)\|x\|_2^2.$$
$$c=(\Omega Q)^T(\Omega v),\qquad v_{\mathrm{new}}=v-Qc,\qquad q_{\mathrm{new}}=\frac{v_{\mathrm{new}}}{\|\Omega v_{\mathrm{new}}\|_2}.$$
$$H=V^TAV,\qquad G=V^TV\succ0,\qquad Hy=\theta Gy,\qquad u=Vy,\qquad \theta=\frac{u^TAu}{u^Tu}.$$
$$\Delta=-U\,\operatorname{diag}\!\left(\frac{1}{\theta_i+\lambda}\right)U^Tg,\qquad \|\Delta\|_2\leq\rho,$$

Mathematical statement

Let V = [v_1, ..., v_k] be a block of parameter-space directions in R^n, and let Omega be a random sketching matrix in R^(s x n), with s much smaller than n. The paper defines the sketched inner product as <x,y>_Omega = (Omega x)^T(Omega y). Its randomized Gram-Schmidt step computes c = (Omega Q)^T(Omega v), then subtracts Qc from v, where Q contains previously accepted directions. The sketch is intended to satisfy the subspace-embedding inequality (1-epsilon)||x||_2^2 <= ||Omega x||_2^2 <= (1+epsilon)||x||_2^2 on the retained subspace, which preserves conditioning approximately. Since the resulting basis is not necessarily Euclidean-orthonormal, form H = V^T A V and G = V^T V, where A is a symmetric Hessian, Fisher, or generalized curvature operator. Solve the definite generalized eigenproblem H y = theta G y with G positive definite. The Ritz vector is u = V y and has Rayleigh quotient theta = (u^T A u)/(u^T u). In the optimizer, Hessian-vector products provide AV, while G and H are computed by parameter-space reductions.

Implementation notes

Integrate this into a block curvature optimizer between gradient computation and the parameter update. Let the flattened parameter dimension be n, choose block size k such as 8 or 16, sketch size s such as 4k to 8k, and a fixed sparse signed sketch Omega implemented as a hash-and-sign map. Generate candidate directions from the current gradient, momentum direction, and additional Hessian-vector products on random probes or retained directions. Maintain both each full direction and its sketch. Pseudocode: initialize Q and Z as empty; for each candidate v, compute z = Omega v, c = Z transpose times z, replace v by v minus Qc and z by z minus Zc, reject the direction when the sketched residual norm is below a threshold, otherwise append v divided by its sketched norm to Q and z divided by its norm to Z. Compute W_j = A v_j using automatic-differentiation Hessian-vector products or Fisher-vector products. Form H_ij = v_i transpose times W_j and G_ij = v_i transpose times v_j, symmetrize both matrices, and solve H Y = G Y Theta. Set U = VY and use the selected modes in the damped update Delta = -U diag(1/(theta_i + lambda)) U transpose g. Increase lambda until the step norm is at most rho. Compute the sketch operations from the paper directly; estimate embedding quality by testing the ratio ||Omega Vx|| divided by ||Vx|| for random x. First test on a two-layer MLP or ResNet-18 on CIFAR-10 against SGD, AdamW, and the same block optimizer with full Gram-Schmidt, matching Hessian-vector-product counts. Record wall-clock optimizer overhead, loss decrease per second, validation accuracy, condition number of G, and generalized-eigensolver failures. A positive result is at least 10 percent lower orthogonalization overhead with unchanged accuracy, or faster loss descent at equal wall-clock time.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.