Sketched curvature-subspace optimizer
Implementation & benchmark of arXiv:2608.24529 — Randomized Block Davidson Eigensolvers for Plane-Wave Density-Functional Theory
Source paper: Randomized Block Davidson Eigensolvers for Plane-Wave Density-Functional Theory arXiv:2608.24529 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Construct a block of gradient, preconditioned-gradient, or Hessian-vector-product directions without performing full-dimensional Gram-Schmidt. Use a random sketch to orthogonalize the block cheaply, then solve a small generalized eigenproblem using the true parameter-space overlap matrix so the extracted curvature modes are accurate for the generated subspace. Use the selected curvature modes to form a damped or trust-region optimizer step.
Formulas
Mathematical statement
Let V = [v_1, ..., v_k] be a block of parameter-space directions in R^n, and let Omega be a random sketching matrix in R^(s x n), with s much smaller than n. The paper defines the sketched inner product as <x,y>_Omega = (Omega x)^T(Omega y). Its randomized Gram-Schmidt step computes c = (Omega Q)^T(Omega v), then subtracts Qc from v, where Q contains previously accepted directions. The sketch is intended to satisfy the subspace-embedding inequality (1-epsilon)||x||_2^2 <= ||Omega x||_2^2 <= (1+epsilon)||x||_2^2 on the retained subspace, which preserves conditioning approximately. Since the resulting basis is not necessarily Euclidean-orthonormal, form H = V^T A V and G = V^T V, where A is a symmetric Hessian, Fisher, or generalized curvature operator. Solve the definite generalized eigenproblem H y = theta G y with G positive definite. The Ritz vector is u = V y and has Rayleigh quotient theta = (u^T A u)/(u^T u). In the optimizer, Hessian-vector products provide AV, while G and H are computed by parameter-space reductions.
Implementation notes
Integrate this into a block curvature optimizer between gradient computation and the parameter update. Let the flattened parameter dimension be n, choose block size k such as 8 or 16, sketch size s such as 4k to 8k, and a fixed sparse signed sketch Omega implemented as a hash-and-sign map. Generate candidate directions from the current gradient, momentum direction, and additional Hessian-vector products on random probes or retained directions. Maintain both each full direction and its sketch. Pseudocode: initialize Q and Z as empty; for each candidate v, compute z = Omega v, c = Z transpose times z, replace v by v minus Qc and z by z minus Zc, reject the direction when the sketched residual norm is below a threshold, otherwise append v divided by its sketched norm to Q and z divided by its norm to Z. Compute W_j = A v_j using automatic-differentiation Hessian-vector products or Fisher-vector products. Form H_ij = v_i transpose times W_j and G_ij = v_i transpose times v_j, symmetrize both matrices, and solve H Y = G Y Theta. Set U = VY and use the selected modes in the damped update Delta = -U diag(1/(theta_i + lambda)) U transpose g. Increase lambda until the step norm is at most rho. Compute the sketch operations from the paper directly; estimate embedding quality by testing the ratio ||Omega Vx|| divided by ||Vx|| for random x. First test on a two-layer MLP or ResNet-18 on CIFAR-10 against SGD, AdamW, and the same block optimizer with full Gram-Schmidt, matching Hessian-vector-product counts. Record wall-clock optimizer overhead, loss decrease per second, validation accuracy, condition number of G, and generalized-eigensolver failures. A positive result is at least 10 percent lower orthogonalization overhead with unchanged accuracy, or faster loss descent at equal wall-clock time.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.