# Space-Time Onsager Optimizer

- ID: 1816
- Canonical URL: https://synthcore.org/idea/1816/space-time-onsager-optimizer
- API JSON: https://synthcore.org/api/idea/1816.json
- API Markdown: https://synthcore.org/api/idea/1816.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.04920](https://arxiv.org/abs/2608.04920)
- Category: dynamics
- Solves: stability, speedup, scalability
- ML areas: optimizer, training-dynamics, transformer, ssm
- Math tags: statistical-mechanics, stochastic-processes, control-theory, dynamical-systems, linear-algebra
- Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10

## Idea description

Replace an instantaneous diagonal optimizer with a causal convolution of recent gradients, where cross-layer or cross-module gradient correlations define a finite-memory Onsager response matrix. Estimate the response at several parameter-block pairs and lags, integrate it to obtain a finite-time transport matrix, and use its regularized inverse or symmetric part to precondition the update. This targets optimization regimes in which gradients propagate between blocks with measurable delay, such as deep transformers, recurrent networks, and diffusion models.

## Mathematical statement

Let $g_a(k)\in\mathbb{R}^{d_a}$ be the gradient of the loss at optimization step $k$ for parameter block $a$, and let $\tau$ denote a nonnegative lag. The paper's spatially resolved response kernel $\mathbf{R}_{ab}(\mathbf r,\mathbf r',t)$ is adapted to a block-and-lag kernel $R_{ab}(\tau)$ estimated from gradient fluctuations. Its local-global and global-global reductions become $K_{ab}(\tau)=\sum_b R_{ab}(\tau)$ and $C_{ab}(\tau)=\sum_a K_{ab}(\tau)$, while the finite-time integrated response is $\mathcal M_{ab}(T)=\sum_{\tau=0}^{T}K_{ab}(\tau)$. The long-time Onsager matrix is $M_{ab}=\lim_{T\to\infty}\mathcal M_{ab}(T)$ when correlations are summable. Use a symmetrized, ridge-regularized matrix $P=(M+M^{\mathsf T})/2+\lambda I$ as a preconditioner, or use its stable inverse square root. For a locally quadratic loss with Hessian $H$, the update $\theta_{k+1}=\theta_k-\eta P^{-1}g_k$ is linearly stable when $0<\eta<2/\lambda_{\max}(P^{-1/2}HP^{-1/2})$. Persistent and rapidly relaxing response modes correspond to different learned gradient-memory times.

## Key formulas

- $$K_{ab}(\mathbf r,t)=\int_V R_{ab}(\mathbf r,\mathbf r',t)\,\mathrm d\mathbf r',\qquad \mathcal M_{ab}(\mathbf r,t)=\int_0^t K_{ab}(\mathbf r,s)\,\mathrm ds,\qquad M_{ab}(\mathbf r)=\lim_{t\to\infty}\mathcal M_{ab}(\mathbf r,t).$$
- $$C_{ab}(t)=\int_V K_{ab}(\mathbf r,t)\,\mathrm d\mathbf r,\qquad \mathcal L_{ab}(t)=\int_0^t C_{ab}(s)\,\mathrm ds,\qquad L_{ab}=\lim_{t\to\infty}\mathcal L_{ab}(t).$$
- $$\widehat R_{ab}(\tau)=\frac{1}{B-\tau}\sum_{k=\tau+1}^{B}\bigl(g_a(k)-\bar g_a\bigr)\bigl(g_b(k-\tau)-\bar g_b\bigr)^{\mathsf T},\qquad \widehat M_{ab}(T)=\sum_{\tau=0}^{T}w_\tau\widehat R_{ab}(\tau).$$
- $$\theta_{k+1}=\theta_k-\eta P^{-1}\widetilde g_k,\qquad \widetilde g_{a,k}=g_{a,k}+\alpha\sum_{\tau=1}^{T}w_\tau\widehat R_{ab}(\tau)g_{b,k-\tau},\qquad P=\frac{\widehat M+\widehat M^{\mathsf T}}{2}+\lambda I.$$

## Implementation notes

1. Integration point: partition the network parameters into blocks, such as attention projections, MLP projections, normalization parameters, or recurrent state matrices. Maintain a FIFO queue of the last $T$ block gradients after flattening each block and optionally projecting it to a small sketch dimension with a fixed random matrix. Estimate lagged cross-block covariances online using exponential averaging, avoiding full parameter-sized matrices. Begin with one scalar covariance per block pair or low-rank factors; use blockwise diagonal or Kronecker approximations for scalability. 2. Pseudocode: compute gradients $g_{a,k}$; update means $\bar g_a$ and covariance estimates $R_{ab}(\tau)\leftarrow(1-\beta)R_{ab}(\tau)+\beta(g_{a,k}-\bar g_a)(g_{b,k-\tau}-\bar g_b)^{\mathsf T}$; form $M=\sum_{\tau=0}^{T}w_\tau R(\tau)$; set $P=(M+M^{\mathsf T})/2+\lambda I$; solve $P d_k=g_k$ with conjugate gradients or a blockwise inverse; update $\theta\leftarrow\theta-\eta d_k$. A safer variant adds the causal memory term in the fourth formula and clips its norm. 3. Computed from the paper's construction: the kernel hierarchy, finite-time integration, and separation of transient from persistent modes. Estimated empirically: gradient covariances, correlation decay, memory horizon $T$, and the largest eigenvalue of $P^{-1/2}HP^{-1/2}$ using Hessian-vector products. Use backtracking if the quadratic model increases or an estimated spectral radius exceeds one. 4. First cheap experiment: train a 12-layer GPT-style transformer on a small character-level WikiText subset, comparing AdamW, block-diagonal Onsager preconditioning, and the causal-memory variant at matched compute. Use 8 to 16 blocks, $T\in\{4,8,16,32\}$, and low-rank gradient sketches. Measure gradient cross-correlations, loss autocorrelation, wall-clock convergence, and Hessian-vector spectral estimates. The quantitative prediction is that $\widehat M(T)$ plateaus once $T$ exceeds the dominant gradient-correlation time; larger $T$ should provide no further conditioning gain and may destabilize training. On a locally quadratic probe, the measured maximum stable learning rate should agree within 20% with $\eta_c=2/\lambda_{\max}(P^{-1/2}HP^{-1/2})$. Removing cross-block lag terms should eliminate most improvement when off-diagonal correlations are weak, but should significantly reduce convergence time when they are strong.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a reproducible two-block Space-Time Onsager toy probe in onsager_probe.py. The exact finite-memory kernel matched the predicted tail q^(T+1) to 2e-10 relative error, and the measured quadratic stability transition occurred between 0.99 and 1.01 times the predicted eta_c=0.96555. Cross-block preconditioning showed no gain at rho=0 and increasingly strong conditioning/convergence gains as rho rose, with idealized equal-step speedups from 6x at rho=0.2 to 180x at rho=0.95. This establishes the mechanism in a controlled quadratic setting, but not a practical optimizer win.

### Mechanism check

- Verdict: Built a reproducible two-block Space-Time Onsager toy probe in onsager_probe.py. The exact finite-memory kernel matched the predicted tail q^(T+1) to 2e-10 relative error, and the measured quadratic stability transition occurred between 0.99 and 1.01 times the predicted eta_c=0.96555. Cross-block preconditioning showed no gain at rho=0 and increasingly strong conditioning/convergence gains as rho rose, with idealized equal-step speedups from 6x at rho=0.2 to 180x at rho=0.95. This establishes the mechanism in a controlled quadratic setting, but not a practical optimizer win.
- Confidence: 8/10
- Limitations: No transformer or neural-network training was run. The cross-block comparison uses an idealized response matrix equal to the quadratic Hessian rather than an online learned covariance preconditioner. Finite-sample empirical covariance tails were inaccurate when q^(T+1) was extremely small, despite the exact-kernel prediction being confirmed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_onsager.py](https://synthcore.org/code/651/bench_onsager.py)
- [bench_report.json](https://synthcore.org/code/651/bench_report.json)
- [onsager_probe.py](https://synthcore.org/code/651/onsager_probe.py)
- [report.md](https://synthcore.org/code/651/report.md)
- [report_bench_2026-08-31T210231.md](https://synthcore.org/code/651/report_bench_2026-08-31T210231.md)
- [results.json](https://synthcore.org/code/651/results.json)
- [Download all files as ZIP](https://synthcore.org/download/651)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.