# Critical Cross-Layer Weight Sharing

- ID: 2716
- Canonical URL: https://synthcore.org/idea/2716/critical-cross-layer-weight-sharing
- API JSON: https://synthcore.org/api/idea/2716.json
- API Markdown: https://synthcore.org/api/idea/2716.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.23944](https://arxiv.org/abs/2608.23944)
- Category: architecture
- Solves: stability, memory, scalability
- ML areas: transformer, rnn, ssm, training-dynamics
- Math tags: random-matrix, spectral-theory, statistics, dynamical-systems, linear-algebra
- Ratings: usefulness 7/10; difficulty 6/10; novelty 8/10

## Idea description

Construct deep or recurrent networks whose layer weights are correlated across depth with a prescribed power-law covariance, rather than either fully tying or fully independently sampling layers. The paper predicts two usable design boundaries: \(\gamma=1/2\) for divergence of correlation-induced fourth moments and \(\gamma=1\) for loss of summable-correlation flatness.

## Mathematical statement

Let \(W^{(\ell)}\) denote the weight matrix at depth \(\ell\), with independent rows but cross-layer covariance \(\operatorname{Cov}(W^{(\ell)}_{ij},W^{(m)}_{ik})\propto\delta_{jk}c_{|\ell-m|}\), where \(c_t\asymp t^{-\gamma}\). The paper's long-range-correlation mechanism implies that the squared-correlation accumulation \(\sum_t c_t^2\) changes scaling at \(\gamma=1/2\), while the absolute-correlation accumulation \(\sum_t|c_t|\) is finite only for \(\gamma>1\). Use \(\gamma>1\) for a conservative regime, \(1/2<\gamma\le1\) for a controlled long-memory regime, and \(\gamma\le1/2\) as a deliberately stress-tested critical regime.

## Key formulas

- $$\operatorname{Cov}\!\left(W^{(\ell)}_{ij},W^{(m)}_{ik}\right)=\frac{\sigma^2}{d}\,\delta_{jk}\,c_{|\ell-m|},\qquad c_t\asymp t^{-\gamma},$$
- $$\sum_{t=1}^{L}c_t^2\asymp\begin{cases}L^{1-2\gamma},&\gamma<\tfrac12,\\ \log L,&\gamma=\tfrac12,\\ 1,&\gamma>\tfrac12,\end{cases}$$
- $$\sum_{t=1}^{L}|c_t|\asymp\begin{cases}L^{1-\gamma},&\gamma<1,\\ \log L,&\gamma=1,\\ 1,&\gamma>1.\end{cases}$$

## Implementation notes

1. Integration point: implement a depth-indexed correlated parameterization for a residual MLP, Transformer block stack, or linear state-space model. Sample a latent Gaussian process \(g_{i,j}^{(\ell)}\) over layer index \(\ell\) for each matrix coordinate, and use it directly as the layer weight or combine it with iid noise: \(W^{(\ell)}=\sqrt{\alpha}\,G^{(\ell)}+\sqrt{1-\alpha}\,E^{(\ell)}\). Use circulant embedding or a Cholesky factor of the \(L\times L\) covariance matrix to sample the power-law process. 2. Pseudocode: choose \(\gamma\), construct \(C_{\ell m}=c_{|\ell-m|}\), sample each coordinate's vector across depth from \(\mathcal N(0,C)\), normalize every layer to identical Frobenius variance, and run the network. During training, log adjacent-layer covariance, the end-to-end Jacobian singular values, and residual-branch norms. 3. Taken from the paper: the two correlation accumulation thresholds and their different roles. Estimated empirically: whether the neural Jacobian or gradient covariance inherits these scalings; this inheritance is the falsifiable transfer hypothesis, not a theorem supplied by the paper. 4. First cheap experiment: use 24-layer residual MLPs of width 256 on CIFAR-10 or sequence-copying tasks, comparing iid layers, exact tying, AR(1) sharing, and power-law sharing with \(\gamma=0.3,0.5,0.75,1.0,1.25\). Repeat at depths \(L=12,24,48,96\). The prediction is a kink in the depth scaling of the normalized gradient fourth moment near \(\gamma=1/2\), approximately \(L^{1-2\gamma}\) below it, and a change from growing to bounded cumulative layer-correlation near \(\gamma=1\). If these signatures do not occur in Jacobians or gradient covariance, discard the transfer.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Implemented correlated power-law cross-layer Gaussian weights and a residual-MLP Jacobian experiment. For gamma=0.3, observed tail exponents were 0.701 for absolute-correlation accumulation and 0.409 for squared-correlation accumulation, versus predictions 0.7 and 0.4. Critical gamma=0.5 and gamma=1 sums showed logarithmic growth, and empirical fourth moments matched the exact Wick formula within 0.8%. The correlated network had worse depth-128 Jacobian stability than IID weights, so the mechanism manifested but no performance win was demonstrated.

### Mechanism check

- Verdict: Implemented correlated power-law cross-layer Gaussian weights and a residual-MLP Jacobian experiment. For gamma=0.3, observed tail exponents were 0.701 for absolute-correlation accumulation and 0.409 for squared-correlation accumulation, versus predictions 0.7 and 0.4. Critical gamma=0.5 and gamma=1 sums showed logarithmic growth, and empirical fourth moments matched the exact Wick formula within 0.8%. The correlated network had worse depth-128 Jacobian stability than IID weights, so the mechanism manifested but no performance win was demonstrated.
- Confidence: 9/10
- Limitations: Only a toy Gaussian-coordinate and small residual-MLP Jacobian experiment was tested; no CIFAR or sequence training, loss/FLOP comparison, tied or AR(1) baseline, learned mixing coefficient, or large-width/depth study was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.31345
- Paired wins: 6/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_critical_sharing.py](https://synthcore.org/code/1048/bench_critical_sharing.py)
- [bench_report.json](https://synthcore.org/code/1048/bench_report.json)
- [critical_sharing_experiment.py](https://synthcore.org/code/1048/critical_sharing_experiment.py)
- [report.md](https://synthcore.org/code/1048/report.md)
- [report_bench_2026-09-01T194105.md](https://synthcore.org/code/1048/report_bench_2026-09-01T194105.md)
- [results.json](https://synthcore.org/code/1048/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1048)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.