# Nonequilibrium Coupled-Block Noise

- ID: 2727
- Canonical URL: https://synthcore.org/idea/2727/nonequilibrium-coupled-block-noise
- API JSON: https://synthcore.org/api/idea/2727.json
- API Markdown: https://synthcore.org/api/idea/2727.md
- Verification status: unverified
- Source: [arXiv:2608.24980](https://arxiv.org/abs/2608.24980)
- Category: dynamics
- Solves: stability, generalization, sample-efficiency
- ML areas: optimizer, training-dynamics, regularization
- Math tags: statistical-mechanics, stochastic-processes, linear-algebra, dynamical-systems
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Partition a neural network into coupled parameter or activation blocks with distinct effective noise temperatures, and inject Gaussian perturbations whose covariance contains off-diagonal terms induced by the coupling. Unlike standard independent gradient noise, equal-temperature or detached blocks should have negligible cross-correlation, whereas unequal-temperature coupled blocks should exhibit measurable correlated fluctuations.

## Mathematical statement

For a conductor with resistance R, temperature T, and averaging duration Delta t, the paper uses the Johnson-Nyquist relation Var(epsilon) = 2 k_B T R / Delta t, where k_B is Boltzmann's constant and epsilon is the time-averaged electromotive force. Transfer this to m neural-network blocks by assigning block i an effective temperature T_i > 0 and noise scale R_i > 0. Inject xi sampled from a multivariate Gaussian with covariance C, whose diagonal entry C_ii = 2 T_i R_i / Delta t after absorbing k_B into the optimizer scale. The nonequilibrium extension is C_ij != 0 for coupled blocks i != j when T_i != T_j, while C_ij approximately vanishes for detached blocks or equal-temperature equilibrium. A practical parameterization is C_ij = kappa_ij rho(T_i,T_j) sqrt(C_ii C_jj), where kappa_ij is a symmetric coupling coefficient with absolute value below one and rho is zero at equal temperatures and increases with temperature mismatch. Positive semidefiniteness must be enforced by a Cholesky parameterization or projection.

## Key formulas

- $$\operatorname{Var}(\epsilon)=\frac{2k_BTR}{\Delta t}.$$
- $$\xi\sim\mathcal N(0,C),\qquad C_{ii}=\frac{2T_iR_i}{\Delta t},\qquad C_{ij}=\kappa_{ij}\rho(T_i,T_j)\sqrt{C_{ii}C_{jj}}.$$
- $$\operatorname{Corr}(\xi_i,\xi_j)=\frac{C_{ij}}{\sqrt{C_{ii}C_{jj}}},\qquad C_{ij}\to0\ \text{if}\ \kappa_{ij}\to0\ \text{or}\ T_i=T_j.$$

## Implementation notes

1. Integration point: partition parameters or hidden activations into blocks, such as attention projections, MLP projections, and normalization parameters, or simply two groups of residual layers. Apply stochastic perturbations after computing the gradient and before the optimizer update: theta_{t+1} = theta_t - eta g_t + xi_t, where eta is the learning rate and xi_t is sampled jointly across coupled blocks. Begin with two blocks so covariance estimation and debugging remain cheap. Define effective temperatures manually for the first experiment, then estimate them from exponential moving averages of normalized block-gradient variance. 2. Pseudocode: maintain T_i, R_i, and a coupling matrix K; compute C_ii = 2 T_i R_i / Delta t; set C_ij = kappa_ij abs(T_i-T_j)/(T_i+T_j) sqrt(C_ii C_jj); project C to the positive-semidefinite cone or use a Cholesky factor; sample z from a standard normal distribution; set xi = L z with L L^T = C; and update each block theta_i <- theta_i - eta g_i + xi_i. 3. Computed from the paper: the local thermal-noise variance and the qualitative rule that coupling plus temperature mismatch generates nonlocal covariance. Estimated empirically: R_i, effective temperatures, useful coupling strengths, and optimization effects. 4. First cheap experiment: train a two-layer MLP and ResNet-18 on CIFAR-10 with ordinary SGD, independent block noise, and coupled-block noise. Sweep kappa in {0, 0.1, 0.25, 0.5} and temperature mismatch abs(T_1-T_2)/(T_1+T_2) in {0, 0.25, 0.5}. Log empirical cross-correlations of injected noise and block gradients. The quantitative prediction is Corr(xi_1,xi_2) approximately zero at kappa = 0 or equal temperatures, and approximately kappa abs(T_1-T_2)/(T_1+T_2) otherwise; the measured covariance should match the target within 10%. Also sweep eta and compare divergence thresholds. The threshold shift should correlate with the largest eigenvalue of the measured block-update covariance, rather than with total scalar noise variance alone.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.