SynthCore
Unverified 2026

Residual-Curvature Gauss-Newton

Implementation & benchmark of arXiv:2608.05680 — Curvature Residual Geometry in Bregman Regression

Usefulness6/10
Difficulty6/10
Novelty6/10
Optimization StabilitySpeedupOptimizerLossTraining dynamicsMlpConvex analysisOptimizationLinear algebraDynamical systems

Source paper: Curvature Residual Geometry in Bregman Regression arXiv:2608.05680 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the Bregman objective's exact residual-dependent curvature to build a positive-semidefinite Gauss-Newton preconditioner for a neural network's scalar regression head. Negative curvature weights are clipped or damped before solving the update, preserving the original gradient while preventing residual patterns from producing unstable parameter steps.

Formulas

$$D_{\phi}(y\|m)=\phi(y)-\phi(m)-\phi'(m)(y-m),\qquad r=y-m,$$
$$\frac{\partial D_{\phi}(y\|m)}{\partial m}=-\phi''(m)r,\qquad g(\theta)=-\frac{1}{n}X^{\top}a,\quad a_i=\phi''(m_i)r_i,$$
$$H(\theta)=\frac{1}{n}X^{\top}\operatorname{diag}(w_i)X,\qquad w_i=\phi''(m_i)-\phi'''(m_i)r_i,$$
$$G=\frac{1}{n}J^{\top}\operatorname{diag}(w_i^{+})J+\lambda I,\qquad w_i^{+}=\max\{\phi''(m_i)-\phi'''(m_i)r_i,\epsilon\},\qquad \theta_{t+1}=\theta_t-\eta G^{-1}g_t.$$

Mathematical statement

For a scalar prediction m=x^T theta and residual r=y-m, the Bregman divergence is D_phi(y||m)=phi(y)-phi(m)-phi'(m)(y-m). Its derivative with respect to m is partial D/partial m=-phi''(m)r, so for a design matrix X with rows x_i^T the regression gradient is g(theta)=-(1/n)X^T a, where a_i=phi''(m_i)r_i. Differentiating again gives the weighted Gram Hessian H(theta)=(1/n)X^T diag(w_i)X, with w_i=phi''(m_i)-phi'''(m_i)r_i. The term -phi'''(m_i)r_i can make w_i negative even when phi is strongly convex. For a neural network with parameters theta, predictions m_i=f_theta(x_i), and Jacobian rows J_i=partial m_i/partial theta, use the generalized Gauss-Newton approximation G=(1/n)J^T diag(w_i^+)J+lambda I, where w_i^+=max(w_i,epsilon), epsilon>0 is a curvature floor, and lambda>=0 is damping. The update theta<-theta-eta G^{-1}g keeps the exact Bregman gradient but uses only positive residual-aware curvature.

Implementation notes

(1) Integration point: apply this optimizer to the final scalar prediction head of a neural regressor, initially freezing the hidden representation for a clean test and then unfreezing all layers. On each minibatch, compute predictions m_i=f_theta(x_i), residuals r_i=y_i-m_i, the exact Bregman gradient g_i=-phi''(m_i)r_i, and per-example curvature w_i=phi''(m_i)-phi'''(m_i)r_i. Use the network Jacobian J only for the tested parameter block; for larger models use a diagonal approximation or layerwise blocks.

(2) Pseudocode:

for (x,y) in minibatch:
    m = net(x); r = y - m
    loss = mean(phi(y)-phi(m)-phi_prime(m)*r)
    g = autograd(loss, theta)
    w = phi_double_prime(m) - phi_triple_prime(m)*r
    wp = clamp_min(w, eps)
    Gv(v) = J.T @ (wp * (J @ v))/batch_size + lambda*v
    delta = conjugate_gradient(Gv, g, iterations=5-20)
    theta -= eta * delta

The formula for w is the paper's residual-curvature term; the clipped G is the proposed neural-network adaptation.

(3) Compute phi'', phi''' analytically for the chosen potential. Estimate Gv with Jacobian-vector and vector-Jacobian products; do not materialize J. Track q=mean(w<0), the minimum w, and CG residuals. Tune epsilon and lambda on a logarithmic grid, and optionally use backtracking when the minibatch loss rises.

(4) First experiment: train a two-layer MLP on a synthetic heteroscedastic regression task and a small UCI regression dataset using phi(u)=u^2/2+alpha*u^4/4, comparing AdamW, ordinary damped Gauss-Newton, and this clipped residual-curvature method at matched wall-clock time and FLOPs. Test alpha in {0.01,0.1,1}. Success means fewer divergence events, lower loss at equal updates, or faster loss decrease on batches with large q. Verify that gains disappear at alpha=0, where the method reduces to ordinary least-squares curvature.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.

Related ideas