# Correlated Long-Range Residual Mixer

- ID: 3164
- Canonical URL: https://synthcore.org/idea/3164/correlated-long-range-residual-mixer
- API JSON: https://synthcore.org/api/idea/3164.json
- API Markdown: https://synthcore.org/api/idea/3164.md
- Verification status: unverified
- Source: [arXiv:2609.04091](https://arxiv.org/abs/2609.04091)
- Category: architecture
- Solves: accuracy, stability, scalability
- ML areas: transformer, attention, ssm, training-dynamics, initialization
- Math tags: statistical-mechanics, spectral-theory, random-matrix, dynamical-systems, harmonic-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace an iid local or randomly sparse residual mixer with a distance-decaying long-range operator whose edge magnitudes are correlated through a shared latent Gaussian field. The paper predicts that these correlations qualitatively change low-energy spectral scaling and increase multiscale information propagation relative to iid long-range weights. Apply the operator as a spectrally normalized residual block so that the benefit comes from correlated scale coverage rather than uncontrolled amplification.

## Mathematical statement

The paper studies effective Majorana Hamiltonians with random long-range hopping terms, schematically \(H=i\sum_{i<j}t_{ij}\gamma_i\gamma_j\), where \(\gamma_i\) are Majorana operators and hopping magnitudes \(|t_{ij}|\) decay algebraically with distance while possessing nontrivial correlations. Its finite-size signatures are \(\Delta(L)\propto L^{-z}\) with \(1<z<2\), and entanglement \(S(L)\propto[\ln L]^2\) for correlated hoppings, versus approximately \(S(L)\propto\ln L\) for uncorrelated hoppings. Transfer this by defining a neural mixer \(M\) with \(M_{ij}=\sigma_i\sigma_j\epsilon_{ij}(1+|i-j|)^{-\alpha}\), where \(\alpha>0\) is a distance exponent, \(\epsilon_{ij}\) is a zero-mean sign variable, and \(\sigma_i=\exp(g_i-\operatorname{Var}(g)/2)\) is a positive correlated amplitude field. Generate \(g\) with Fourier power spectrum \(P_g(k)\propto(k+k_0)^{-\beta}\), where \(\beta\) controls correlation strength. Use \(x^{+}=x+\eta M\phi(x)\), with \(\eta\|M\|_2\le c<1\). The transferred predictions are an anomalous mixer spectral-gap exponent \(1<z<2\) and a multiscale participation entropy closer to \([\ln L]^2\) than \(\ln L\).

## Key formulas

- $$H=i\sum_{i<j}t_{ij}\gamma_i\gamma_j,\qquad |t_{ij}|\sim |i-j|^{-\alpha},$$
- $$\Delta(L)\sim L^{-z},\qquad 1<z<2;\qquad S_{\mathrm{corr}}(L)\sim[\ln L]^2,\quad S_{\mathrm{iid}}(L)\sim\ln L,$$
- $$M_{ij}=\epsilon_{ij}\exp\!\left(g_i-\frac{1}{2}\operatorname{Var}(g_i)\right)\exp\!\left(g_j-\frac{1}{2}\operatorname{Var}(g_j)\right)(1+|i-j|)^{-\alpha},$$
- $$x_{\ell+1}=x_\ell+\eta_\ell M\,\phi(x_\ell),\qquad \eta_\ell\|M\|_2\le c<1.$$

## Implementation notes

1. Integration point: implement this as a drop-in replacement for the token-mixing matrix in a residual Transformer, MLP-Mixer, or linear state-space block. For sequence length L, construct distances d(i,j)=|i-j|, sample a low-frequency latent vector g by drawing white noise in Fourier space and multiplying mode k by (k+k0)^(-beta/2), then apply an inverse FFT. Set sigma_i=exp(g_i-0.5*Var(g)), sample a fixed random sign matrix epsilon, multiply by (1+d)^(-alpha), and normalize the operator. 2. Pseudocode: `g=ifft(noise*((abs(k)+k0)**(-beta/2))).real; sigma=exp(g-0.5*var(g)); M=epsilon*(sigma[:,None]*sigma[None,:])/(1+dist)**alpha; M=M/spectral_norm(M); x=x+eta*M@gelu(W@x)`. Estimate the norm using 3-10 power iterations and set eta=c/(norm+1e-8), initially c=0.5. 3. Computed from the paper: use algebraic distance decay and compare correlated against iid amplitudes. Estimated empirically: spectral gap, singular-value density, gradient norms, and a propagation-entropy proxy. Define the gap as 1-s2/s1, where s1 and s2 are the two largest singular values, or use the smallest nonzero eigenvalue of M^T M. 4. First cheap experiment: train 4-layer residual token mixers on CIFAR-10 patches or a character-level next-token task, comparing local mixing, iid long-range weights, and correlated long-range weights at matched parameter count and spectral norm. Sweep L in {64,128,256,512} and beta in {0,0.5,1,2}. The falsifiable prediction is that correlated models have a fitted finite-size exponent z between 1 and 2, while iid controls do not show the same anomalous scaling; their participation entropy should fit a*(log L)^2+b with lower residual error than a*log L+b for beta at least 1. Training should remain stable when eta*||M||2 <= 0.5, while removing normalization should cause exploding residual Jacobians near eta*||M||2 approximately 1.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.