# Sparse Lyapunov Search for Safe Optimizer Hyperparameters - ID: 592 - Canonical URL: https://synthcore.org/idea/592/sparse-lyapunov-search-for-safe-optimizer-hyperparameters - API JSON: https://synthcore.org/api/idea/592.json - API Markdown: https://synthcore.org/api/idea/592.md - Verification status: mechanism_failed - Source: [arXiv:2607.08753](https://arxiv.org/abs/2607.08753) - Category: optimization - Solves: stability, generalization - ML areas: optimizer, training-dynamics, initialization - Math tags: optimization, convex-analysis, linear-algebra - Ratings: usefulness 6/10; difficulty 7/10; novelty 7/10 ## Idea description Use the paper's certificate-sparsification procedure to search for a small Lyapunov proof of an optimizer's contraction on local strongly convex quadratic models. The active interpolation inequalities and resulting sparse Lyapunov coefficients become a data-driven rule for limiting learning rate and momentum per layer or parameter block, instead of relying only on global heuristics. ## Mathematical statement For a function that is simultaneously mu-strongly convex and L-smooth, the paper represents each interpolation condition between points (x_i,g_i) and (x_j,g_j) as H_mu,L(x_i,x_j)=+tr(GQ_{i,j}), where G is a positive semidefinite Gram matrix of displacement and gradient vectors, s contains function-value differences, q_{i,j}=e_i-e_j, and Q_{i,j} is defined by tr(GQ_{i,j})=--(1/(2(L-mu)))||g_i-g_j-mu(x_i-x_j)||^2-(mu/2)||x_i-x_j||^2. A performance-estimation proof is an SDP over G and s, with objective , interpolation constraints H_mu,L(x_i,x_j)>=0, and linear equalities encoding optimizer iterations. The paper then performs a second optimization over nonnegative dual multipliers lambda_{ij}, minimizing a weighted l1 cost sum_{ij} w_{ij} lambda_{ij} while retaining a valid certificate. The sparse support identifies the inequalities essential for the Lyapunov/convergence argument. In the ML adaptation, mu and L are estimated from Hessian-vector products or gradient secant pairs, and the sparse certificate is used to choose a safe per-block learning rate eta and momentum beta. ## Key formulas - $$\operatorname{tr}(GQ_{i,j})=-\langle g_j,x_i-x_j\rangle-\frac{1}{2(L-\mu)}\left\lVert g_i-g_j-\mu(x_i-x_j)\right\rVert^2-\frac{\mu}{2}\left\lVert x_i-x_j\right\rVert^2,$$ - $$H_{\mu,L}(x_i,x_j)=\langle s,q_{i,j}\rangle+\operatorname{tr}(GQ_{i,j})\ge 0,$$ - $$\underset{s\in\mathbb{R}^{2},\,G\in\mathbb{S}_{+}^{3}}{\operatorname{maximize}}\;\langle s,q_P\rangle\quad\text{subject to interpolation inequalities and linear algorithm constraints},$$ - $$\min_{\lambda\ge 0}\;\sum_{(i,j)}w_{ij}|\lambda_{ij}|\quad\text{subject to the resulting dual SDP certificate remaining valid and proving }\mathcal{P}_f\le \rho\,\mathcal{I}_f.$$ ## Implementation notes (1) Exact integration point: build an offline certificate-search tool around a momentum-SGD update and use its output online as a per-layer learning-rate and momentum ceiling. For each layer or optimizer parameter block, collect a warmup history of parameter displacements and gradient displacements, then approximate the local objective by a quadratic with curvature interval [mu,L]. (2) Pseudocode: collect K pairs (delta_x, delta_g); estimate L=max_k ||delta_g_k||/max(||delta_x_k||,epsilon) and mu=max(0,min_k /max(||delta_x_k||^2,epsilon)); enumerate a short trajectory x_0,...,x_N under candidate eta and beta; create Gram variables G and function-value variables s; impose G positive semidefinite, H_mu,L(x_i,x_j)>=0 using the displayed interpolation formula, and trajectory equations x_{k+1}=x_k-eta g_k+beta(x_k-x_{k-1}); solve the performance SDP for contraction ratio rho. Then solve the weighted l1 re-optimization over nonnegative dual multipliers lambda_ij, minimizing sum w_ij lambda_ij while preserving rho<=rho_target. Binary-search the largest eta and beta for which a sparse certificate exists. During training, clip the proposed eta_layer and beta_layer to these certified values. (3) Computed from the paper's mathematics: the Gram SDP, interpolation residuals, dual multipliers, active inequality set, and certified contraction bound. Estimated empirically: mu, L, rho_target, and the validity horizon of the local quadratic approximation; smooth curvature estimates with an exponential moving average and recompute certificates every M updates. (4) First cheap experiment: train a 2-layer MLP and a small ResNet on MNIST or CIFAR-10 using momentum SGD, comparing global cosine-decay tuning with certificate-based per-layer ceilings at equal batch size and FLOPs. Log training loss, validation loss, gradient norms, estimated eta times L, loss spikes, and divergence frequency. Success means fewer instability events and faster loss descent during the first 20 percent of training without reducing final validation accuracy; a stronger result is reaching the same validation loss with fewer warmup steps. ## Verification - Status: mechanism_failed - Mechanism evidence: yes - Mechanism confirmed: no - Verdict: Built a compact interpolation-residual verifier and curvature-based sparse certificate proxy for blockwise momentum SGD. The interpolation formula behaved as claimed: valid [mu,L] pairs had nonnegative residuals, while an underestimated L produced a negative residual. However, the secant-based curvature estimates were sometimes nonconservative, causing certificates with endpoint spectral radius above 1 and catastrophic divergence in 5/10 trials; the claimed safe optimizer behavior was therefore not reproducibly observed. ### Mechanism check - Verdict: Built a compact interpolation-residual verifier and curvature-based sparse certificate proxy for blockwise momentum SGD. The interpolation formula behaved as claimed: valid [mu,L] pairs had nonnegative residuals, while an underestimated L produced a negative residual. However, the secant-based curvature estimates were sometimes nonconservative, causing certificates with endpoint spectral radius above 1 and catastrophic divergence in 5/10 trials; the claimed safe optimizer behavior was therefore not reproducibly observed. - Confidence: 9/10 - Limitations: This MVP uses an endpoint spectral-radius proxy rather than the paper's full Gram SDP and dual weighted-l1 sparsification. It tests only small diagonal quadratic blocks, not neural-network training, GPU execution, online certificate recomputation, or robust confidence bounds for curvature estimation. ## Artifacts - [evaluate.py](https://synthcore.org/code/1402/evaluate.py) - [report.md](https://synthcore.org/code/1402/report.md) - [sparse_lyapunov.py](https://synthcore.org/code/1402/sparse_lyapunov.py) - [Download all files as ZIP](https://synthcore.org/download/1402) ## Disclaimer AI-generated research hypothesis, automatically tested. Not peer-reviewed.