Sparse Lyapunov Search for Safe Optimizer Hyperparameters

(1) Exact integration point: build an offline certificate-search tool around a momentum-SGD update and use its output online as a per-layer learning-rate and momentum ceiling. For each layer or optimizer parameter block, collect a warmup history of parameter displacements and gradient displacements, then approximate the local objective by a quadratic with curvature interval [mu,L]. (2) Pseudocode: collect K pairs (delta_x, delta_g); estimate L=max_k ||delta_g_k||/max(||delta_x_k||,epsilon) and mu=max(0,min_k <delta_g_k,delta_x_k>/max(||delta_x_k||^2,epsilon)); enumerate a short trajectory x_0,...,x_N under candidate eta and beta; create Gram variables G and function-value variables s; impose G positive semidefinite, H_mu,L(x_i,x_j)>=0 using the displayed interpolation formula, and trajectory equations x_{k+1}=x_k-eta g_k+beta(x_k-x_{k-1}); solve the performance SDP for contraction ratio rho. Then solve the weighted l1 re-optimization over nonnegative dual multipliers lambda_ij, minimizing sum w_ij lambda_ij while preserving rho<=rho_target. Binary-search the largest eta and beta for which a sparse certificate exists. During training, clip the proposed eta_layer and beta_layer to these certified values. (3) Computed from the paper's mathematics: the Gram SDP, interpolation residuals, dual multipliers, active inequality set, and certified contraction bound. Estimated empirically: mu, L, rho_target, and the validity horizon of the local quadratic approximation; smooth curvature estimates with an exponential moving average and recompute certificates every M updates. (4) First cheap experiment: train a 2-layer MLP and a small ResNet on MNIST or CIFAR-10 using momentum SGD, comparing global cosine-decay tuning with certificate-based per-layer ceilings at equal batch size and FLOPs. Log training loss, validation loss, gradient norms, estimated eta times L, loss spikes, and divergence frequency. Success means fewer instability events and faster loss descent during the first 20 percent of training without reducing final validation accuracy; a stronger result is reaching the same validation loss with fewer warmup steps.