Curvature-Defect Step Control
Implementation & benchmark of arXiv:2608.11486 — Stability of Finite-Batch Particle Mean-Field Variational Inference Beyond Strong Convexity
Source paper: Stability of Finite-Batch Particle Mean-Field Variational Inference Beyond Strong Convexity arXiv:2608.11486 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add a cheap directional curvature-defect estimator to an SGD or AdamW optimizer and shrink the step size only when the local gradient field loses the nominal contraction margin. Unlike a Hessian-norm trust-region rule, this directly measures the quantity that appears as additive instability in the Euler coupling estimate.
Formulas
Mathematical statement
For a differentiable potential V and nominal curvature parameter alpha > 0, the paper defines the curvature defect between x and y as d_alpha(x,y) = [alpha ||x-y||^2 - <grad V(x)-grad V(y), x-y>]_+, where [z]_+ = max(z,0). If grad V is L-Lipschitz and x^+ = x - h grad V(x), y^+ = y - h grad V(y), expanding the squared distance and using <grad V(x)-grad V(y),x-y> >= alpha||x-y||^2 - d_alpha(x,y) gives the one-step bound ||x^+-y^+||^2 <= (1 - 2h alpha + h^2 L^2)||x-y||^2 + 2h d_alpha(x,y). Here h is the learning rate, L is a local or global gradient-Lipschitz estimate, and d_alpha is the additive loss of contraction. The proposed optimizer uses the normalized defect r = d_alpha/(alpha||x-y||^2 + epsilon) as a directly measurable instability score and reduces h when r is large.
Implementation notes
Integrate this into the learning-rate line of SGD or AdamW, after computing the ordinary minibatch update but before applying it. Maintain a cheap probe parameter vector tilde_theta_t that shares the current weights but receives a small normalized perturbation: tilde_theta_t = theta_t + rho u_t, where u_t is a fresh Rademacher vector normalized to unit norm. Compute two gradients on the same minibatch when possible, g_t = grad_theta L(theta_t) and tilde_g_t = grad_theta L(tilde_theta_t). Estimate the directional defect with d_t = max(0, alpha rho^2 - dot(tilde_g_t-g_t, tilde_theta_t-theta_t)). Set r_t = d_t/(alpha rho^2+epsilon), smooth r_t with an exponential moving average, and update the base learning rate by h <- clip(h/(1+gamma r_t), h_min, h_max). For AdamW, apply this multiplier to the bias-corrected adaptive step before decoupled weight decay; do not alter the Adam moments. Estimate alpha from a desired nominal contraction margin, for example alpha = 0.01 times a running median of positive directional curvature dot(tilde_g-g,tilde_theta-theta)/rho^2, clipped to a positive range. Estimate L from the same secant as L_t = ||tilde_g-g||/rho and optionally enforce h <= 0.9*2*alpha/(L_t^2+2*alpha) when alpha is positive. The defect formula and additive-error interpretation come from the paper; secant estimation, gamma, rho, and clipping are empirical choices. First test on CIFAR-10 with a six-layer CNN and a small ViT, comparing SGD and AdamW against cosine decay at equal gradient evaluations. Log r_t, learning rate, gradient norm, loss spikes, and time per step. Success means fewer divergence or loss-spike runs and faster loss decrease at equal final learning rate. A negative result is increased gradient-computation cost without improved stability or time-to-accuracy.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.