SynthCore
Unverified 2026

Rational Jacobi Curvature Preconditioner

Implementation & benchmark of arXiv:2607.29386 — Rational Jacobi Rotations and the Complexity of Approximating Mixed Integer Quadratic Programming

Usefulness6/10
Difficulty6/10
Novelty6/10
Optimization StabilitySpeedupAccuracyOptimizerTraining dynamicsQuantizationLinear algebraNumerical analysisOptimizationSpectral theory

Source paper: Rational Jacobi Rotations and the Complexity of Approximating Mixed Integer Quadratic Programming arXiv:2607.29386 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace an ordinary dense or floating-point eigendecomposition of small Hessian or Fisher blocks with a sequence of rational Jacobi rotations. The rotations preserve Euclidean norms and can be stored using fixed-point coefficients, while approximately diagonalizing curvature so the optimizer can use separate coordinate-wise step sizes. This is especially relevant to low-precision training and blocks with mixed-sign curvature.

Formulas

$$\begin{bmatrix}\hat{b}_{pp}&\hat{b}_{pq}\\ \hat{b}_{qp}&\hat{b}_{qq}\end{bmatrix}=\begin{bmatrix}\hat{c}&\hat{s}\\-\hat{s}&\hat{c}\end{bmatrix}^{\mathsf{T}}\begin{bmatrix}a_{pp}&a_{pq}\\a_{qp}&a_{qq}\end{bmatrix}\begin{bmatrix}\hat{c}&\hat{s}\\-\hat{s}&\hat{c}\end{bmatrix}.$$
$$b_{pq}=0=a_{pq}\left(c^{2}-s^{2}\right)+\left(a_{pp}-a_{qq}\right)cs,$$
$$\hat{b}_{pq}=a_{pq}\left(\hat{c}^{2}-\hat{c}^{2}\right)+\left(a_{pp}-a_{qq}\right)\hat{c}\hat{s}.$$
$$c=\frac{1-t^2}{1+t^2},\qquad s=\frac{2t}{1+t^2},\qquad c^2+s^2=1.$$
$$u=Q\,\operatorname{diag}\left(\frac{\operatorname{sign}(d_i)}{|d_i|+\lambda}\right)Q^{\mathsf T}g.$$

Mathematical statement

For a symmetric curvature block A, select coordinates p and q and write its 2-by-2 principal submatrix as [[a_pp,a_pq],[a_qp,a_qq]]. A Jacobi rotation R=[[c,s],[-s,c]] transforms the block to B=R^T A R. Exact elimination of the selected off-diagonal entry requires b_pq=0 with 0=a_pq(c^2-s^2)+(a_pp-a_qq)cs. Instead of generally irrational c and s, parameterize a rational norm-preserving rotation by a rational t: c=(1-t^2)/(1+t^2), s=2t/(1+t^2), which gives c^2+s^2=1 exactly. Choose t from the real Jacobi root and round t to a rational or fixed-point grid. The resulting residual is b_pq=a_pq(c^2-s^2)+(a_pp-a_qq)cs, which is an explicit approximation-error signal. Each rotation is orthogonal, so the ideal transformed matrix has exactly the same inertia, meaning the numbers of positive, zero, and negative eigenvalues are preserved. Accumulating rotations in Q and retaining the transformed diagonal d_i gives a signed damped preconditioner Q diag(sign(d_i)/(abs(d_i)+lambda)) Q^T.

Implementation notes

(1) Integration point: insert a blockwise curvature preconditioner immediately before the AdamW or SGD parameter update. Partition parameters into blocks of 16--64 coordinates, estimate a symmetric curvature matrix A for each block using an empirical Fisher or Hessian-vector products, and refresh the preconditioner every K optimizer steps. Never build a full-model Hessian.

(2) Pseudocode:

for each block A:
    Q = identity
    R = A
    repeat J times:
        (p,q) = argmax_{i != j} abs(R[i,j])
        alpha = R[p,q]
        delta = R[p,p] - R[q,q]
        solve alpha*(1-t*t) + delta*t = 0 for t
        t = round_to_grid(t, step=2**(-bits))
        c = (1-t*t)/(1+t*t)
        s = 2*t/(1+t*t)
        G = identity_with_block(p,q, [[c,s],[-s,c]])
        R = G.T @ R @ G
        Q = Q @ G
    d = diagonal(R)
    z = Q.T @ gradient_block
    u = sign(d) * z / (abs(d) + lambda)
    preconditioned_gradient = Q @ u

Apply the parameter update theta_block -= eta * preconditioned_gradient, with decoupled weight decay if desired. For noisy curvature, use an exponential moving average of A or of d. Clip signed negative-curvature steps to a trust-region norm.

(3) Computed directly from the mathematics: the Jacobi cancellation equation, rational parameterization, orthogonality, accumulated Q, and residual off-diagonal entries. Estimated empirically: A, the largest off-diagonal pair, refresh interval K, damping lambda, and fixed-point precision. Log ||offdiag(R)||_F / ||offdiag(A)||_F and compare it with measured gradient covariance.

(4) First experiment: train a 1--5M parameter MLP and a small ViT on CIFAR-10 using SGD, AdamW, and a float32 block-Jacobi baseline. Use 32-dimensional blocks, J=8 rotations per refresh, K=100 steps, and rotation precision in {4, 8, 12} bits. Compare loss versus optimizer steps, wall-clock time, gradient norms, and final accuracy. The target signal is faster loss decrease at equal FLOPs, fewer exploding updates on indefinite blocks, and less than 1% final-accuracy degradation when rotations use 8-bit fixed-point parameters. Ablate rationalization, negative-curvature signs, and rotation ordering.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.

Related ideas