# Curvature-Density Monitor for Optimization Transitions

- ID: 1580
- Canonical URL: https://synthcore.org/idea/1580/curvature-density-monitor-for-optimization-transitions
- API JSON: https://synthcore.org/api/idea/1580.json
- API Markdown: https://synthcore.org/api/idea/1580.md
- Verification status: unverified
- Source: [arXiv:2607.29170](https://arxiv.org/abs/2607.29170)
- Category: dynamics
- Solves: stability, generalization
- ML areas: training-dynamics, optimizer, regularization
- Math tags: differential-geometry, dynamical-systems, statistics
- Ratings: usefulness 5/10; difficulty 7/10; novelty 8/10

## Idea description

Build a two-dimensional local metric from the neural-network loss along a pair of controlled parameter directions, such as the optimizer velocity and a stochastic-gradient fluctuation direction. Compute both scalar curvature R and curvature density mathcal R = sqrt(|g|) R, then use their different peaks or scaling laws to detect sharp optimization transitions and trigger learning-rate or regularization changes.

## Mathematical statement

For a two-dimensional metric g_ab(z), with coordinates z = (z^1,z^2), determinant g = det(g_ab), and scalar curvature R[g] computed from the metric and its derivatives, define the curvature density mathcal R(z) = sqrt(|g(z)|) R[g](z). The paper reports critical scalings R proportional to t^(-d nu) and mathcal R proportional to t^(-(1+beta)), where t is reduced distance from a critical point, d is spatial dimension, nu is the correlation-length exponent, and beta is the order-parameter exponent. For neural training, t can be defined as normalized distance from a detected optimization transition along update count or cumulative parameter displacement. The transferable prediction is that R and mathcal R have different peak locations or fitted exponents, so they can provide complementary early-warning signals.

## Key formulas

- $$\mathcal{R}(z)=\sqrt{|\det g(z)|}\,R[g](z)$$
- $$R\sim t^{-d\nu},\qquad \mathcal{R}\sim t^{-(1+\beta)}$$
- $$g_{ab}(z)=\mathbb{E}_{\xi}\left[\partial_a\ell(\theta(z);\xi)\,\partial_b\ell(\theta(z);\xi)\right]+\lambda\delta_{ab},\qquad a,b\in\{1,2\}$$
- $$\eta_{k+1}=\eta_k\times\begin{cases}\gamma_{\downarrow},&|\mathcal{R}_k|>\tau_{\mathcal{R}}\ \text{or}\ |R_k|>\tau_R\\1,&\text{otherwise}\end{cases},\qquad 0<\gamma_{\downarrow}<1$$

## Implementation notes

1. Integration point: add a monitoring callback to SGD or Adam every K updates. Define a local chart theta(z1,z2) = theta_k + z1 u_k + z2 v_k, where u_k is the normalized recent parameter-update direction and v_k is a normalized mini-batch-gradient-noise direction orthogonalized against u_k. Evaluate the loss on a small 5 by 5 grid around z = 0. 2. Pseudocode: estimate per-example or mini-batch gradients at each grid point; form g_ab as the average of partial_a loss times partial_b loss plus lambda times the identity; fit a smooth bicubic interpolant to the three independent metric components; use automatic differentiation or finite differences to obtain metric derivatives; compute Christoffel symbols and the two-dimensional scalar curvature R; output mathcal R = sqrt(abs(det(g))) times R. If either observable crosses a threshold, multiply the next learning rate by gamma_down, and restore it after M stable checks. 3. Taken from the paper: the distinction between R and curvature density and their different critical scaling. Estimated empirically: the local metric, derivatives, transition location, and exponents. Use lambda equal to 10^-6 times the median diagonal metric value to avoid singular charts. 4. First cheap experiment: train two-layer MLPs of widths 128 and 256 on MNIST with SGD over a learning-rate sweep. Compare standard SGD, curvature-triggered SGD, and a Hessian-trace monitor. Record loss, gradient norm, sharpness, R, and mathcal R. The predicted signature is a reproducible curvature peak or kink before divergence, with peak locations for R and mathcal R differing by several monitoring intervals. Fit each observable to Q proportional to abs(s - s_c)^(-p); the exponents should differ rather than collapse to one value. The method is promising only if the divergence boundary shifts upward or loss variance decreases by at least 10 percent at equal gradient budget.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.