# Onsager-Coupled Block Optimizer

- ID: 1249
- Canonical URL: https://synthcore.org/idea/1249/onsager-coupled-block-optimizer
- API JSON: https://synthcore.org/api/idea/1249.json
- API Markdown: https://synthcore.org/api/idea/1249.md
- Verification status: unverified
- Source: [arXiv:2607.21742](https://arxiv.org/abs/2607.21742)
- Category: optimization
- Solves: speedup, stability, scalability
- ML areas: optimizer, training-dynamics, mlp
- Math tags: optimization, dynamical-systems, spectral-theory, convex-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Use a symmetric positive-definite, non-diagonal mobility matrix to couple updates of parameter groups, analogous to drag-modified Onsager mobility coupling ionic species. Estimate local block curvature and select the learning rate from the generalized spectrum of mobility times curvature, targeting rapid loss decay without the instability of aggressively scaled diagonal optimizers.

## Mathematical statement

The paper uses the free energy E(c) = integral over Omega of [c_n log(c_n) + c_p log(c_p) + one-half times |gradient phi| squared] dx, with -Laplacian(phi) = c_p - c_n. Its dissipation principle gives velocities proportional to negative chemical-potential gradients; with interspecies drag, the analogous mobility is non-diagonal and concentration-dependent. We transplant the finite-dimensional gradient-flow form d theta/dt = -M(theta) gradient L(theta), where theta is the parameter vector, L is the training loss, and M is symmetric positive definite. For the local quadratic model L(theta + delta) approximately equals L(theta) + g-transpose-delta + one-half delta-transpose-H-delta, the linearized dynamics are d delta/dt = -M H delta, and decay rates are eigenvalues of M-to-the-one-half H M-to-the-one-half. If lambda_min and lambda_max are positive estimates of the extreme eigenvalues, explicit Euler is stable for learning rate eta between zero and 2/lambda_max, and the quadratic contraction factor is minimized by eta = 2/(lambda_min + lambda_max). The adaptation uses a K by K SPD mobility over parameter blocks rather than independent scalar coordinates.

## Key formulas

- E_cl(c) = integral over Omega of [c_n log(c_n) + c_p log(c_p) + (1/2)|gradient phi|^2] dx, with -Laplacian(phi) = c_p - c_n.
- mu_n = log(c_n) + 1 - phi, and mu_p = log(c_p) + 1 + phi.
- d theta/dt = -M(theta) gradient L(theta), with M(theta) = M(theta)-transpose and M(theta) positive definite.
- eta_star = 2/(lambda_min + lambda_max), with 0 < eta < 2/lambda_max, and lambda_i in the eigenvalues of M-to-the-one-half H M-to-the-one-half.

## Implementation notes

(1) Integration point: modify the optimizer update after backpropagation, at the level of K parameter groups rather than individual parameters. Use groups consisting of transformer blocks, MLP layers, or attention projection matrices. Let g_k be the flattened gradient for group k. Maintain an SPD mobility M in R^(K by K) and apply the same coupling coefficient to every tensor in a group: u_k = sum over j of M_kj g_j, followed by theta_k <- theta_k - eta u_k. To control scale, combine the coupling with within-group RMS normalization only after testing raw gradients. (2) Pseudocode: receive gradients g_1 through g_K and exponential-moving second moments v_k. Construct diagonal D with D_kk = 1/(sqrt(v_k) + epsilon). Parameterize M = D-to-the-one-half times (A A-transpose + rho I) times D-to-the-one-half for unconstrained A. Normalize M by M <- M divided by (trace(M)/K), compute u = M g, and update theta <- theta - eta u. Every T steps, estimate block Hessian-vector products H v with Pearlmutter automatic differentiation. Run 3 to 5 Lanczos iterations on the operator v mapped to M-to-the-one-half H M-to-the-one-half v and use the Ritz values to estimate lambda_max and, when affordable, lambda_min. Set eta = alpha times 2/(lambda_min + lambda_max), with alpha between 0.3 and 0.9, clipping at 0.9 times 2/lambda_max. (3) The mathematical guarantee being used is that positive-definite mobility produces a valid dissipative gradient flow; the Hessian-vector products, eigenvalues, and matrix entries are estimated empirically. Reject an update if Cholesky factorization fails, and regularize off-diagonal entries toward zero initially. (4) First experiment: train a six-layer MLP or small ViT on MNIST or CIFAR-10 using SGD, AdamW, and diagonal RMSProp as baselines. Use K = 6 layer groups and compare at equal optimizer overhead and FLOPs. Record training loss versus updates and wall-clock time, gradient-norm oscillation, maximum stable learning rate, and final validation accuracy. Success means faster loss descent at equal steps, a larger stable learning-rate interval, or lower loss at equal wall-clock time after including Lanczos and matrix-coupling overhead.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.