# Noise-prune recurrent weights by covariance-aware retention

- ID: 1844
- Canonical URL: https://synthcore.org/idea/1844/noise-prune-recurrent-weights-by-covariance-aware-retention
- API JSON: https://synthcore.org/api/idea/1844.json
- API Markdown: https://synthcore.org/api/idea/1844.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.05464](https://arxiv.org/abs/2608.05464)
- Category: regularization
- Solves: memory, speedup, accuracy
- ML areas: rnn, ssm, pruning, regularization
- Math tags: probability, statistics, linear-algebra, dynamical-systems
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

Replace magnitude pruning in a trained recurrent network with stochastic pruning probabilities computed from weight magnitudes and the covariance of neuron activities under injected noise. Connections whose endpoints fluctuate in a sign-compatible way receive higher retention probability, while retained weights are rescaled to preserve average recurrent strength. The method uses local weights and activity covariance, avoiding Hessian construction and expensive global saliency optimization.

## Mathematical statement

Let x_i(t) be the activity of neuron i under noisy input, let w_{ij} be the directed weight from neuron j to neuron i, and let C be the activity covariance matrix, with C_{ij}=Cov[x_i,x_j] and C_{ii}=Var[x_i]. The paper defines the retention probability p_{ij} using sign-dependent fluctuation energy: for an excitatory weight w_{ij}>0, p_{ij}=K w_{ij}(C_{ii}+C_{jj}-2C_{ij}); for an inhibitory weight w_{ij}<0, p_{ij}=K|w_{ij}|(C_{ii}+C_{jj}+2C_{ij}). K controls overall density. The two covariance expressions equal Var[x_i-x_j] and Var[x_i+x_j], respectively, so they are nonnegative and measure endpoint fluctuations in the sign-compatible direction. In implementation, clip probabilities to [0,1] using p_{ij}=min(1,Ks_{ij}), where s_{ij} is the relevant score. Sample m_{ij}~Bernoulli(p_{ij}) independently and set the pruned weight to zero when m_{ij}=0. A practical layerwise rescaling is w'_{ij}=m_{ij}w_{ij}R^alpha, where R=sum_{a,b}|w_{ab}|/(sum_{a,b}m_{ab}|w_{ab}|+epsilon), epsilon prevents division by zero, and alpha controls rescaling strength. Alpha=1 preserves total absolute weight approximately; alpha<1 tests the paper's observation that weaker empirical rescaling can outperform the theoretically motivated value.

## Key formulas

- $$p_{ij}=\begin{cases}K w_{ij}\left(C_{ii}+C_{jj}-2C_{ij}\right),&w_{ij}>0\\K|w_{ij}|\left(C_{ii}+C_{jj}+2C_{ij}\right),&w_{ij}<0\end{cases}$$
- $$C_{ii}+C_{jj}-2C_{ij}=\operatorname{Var}(x_i-x_j),\qquad C_{ii}+C_{jj}+2C_{ij}=\operatorname{Var}(x_i+x_j)$$
- $$m_{ij}\sim\operatorname{Bernoulli}(p_{ij}),\qquad \widetilde w_{ij}=m_{ij}w_{ij}\left(\frac{\sum_{a,b}|w_{ab}|}{\sum_{a,b}m_{ab}|w_{ab}|+\varepsilon}\right)^{\alpha}$$
- $$p_{ij}=\min\{1,Ks_{ij}\},\qquad K\ \text{chosen so that}\ \frac{1}{|E|}\sum_{(i,j)\in E}p_{ij}=\rho$$

## Implementation notes

1. Integration point: apply the method to the recurrent matrix W_hh of an Elman RNN, GRU, or leaky state-space model after ordinary pretraining. It can also be repeated in a prune-and-finetune schedule, but initially use one-shot pruning for a clean comparison. 2. Pseudocode: initialize covariance accumulators S=0 and Q=0. For b=1,...,B, run minibatches with small zero-mean Gaussian recurrent or preactivation noise and collect hidden states H_b in R^{T x N}. Update S by the sum of hidden vectors and Q by the sum of outer products. After M total time points, compute mu=S/M and C=Q/M-mu mu^T. For every W[i,j], compute s[i,j]=W[i,j]*(C[i,i]+C[j,j]-2*C[i,j]) when W[i,j]>0; otherwise compute s[i,j]=abs(W[i,j])*(C[i,i]+C[j,j]+2*C[i,j]). Choose K by bisection so the mean of min(1,K*s[i,j]) equals target density rho. Sample mask[i,j] from Bernoulli(min(1,K*s[i,j])), zero masked weights, and compute R=sum(abs(W))/(sum(abs(mask*W))+epsilon). Set W=mask*W*R^alpha and fine-tune. 3. Quantities supplied by the mathematics are the sign-dependent score, Bernoulli retention rule, and strength rescaling. Estimate C from noisy hidden-state samples; select rho, noise amplitude, alpha, and B empirically. Use separate covariance-estimation and validation batches. Preserve self-connections or enforce a minimum in-degree if pruning destabilizes the recurrent dynamics. 4. First experiment: train 64- or 128-unit RNNs on Copy Memory, adding, and sequential MNIST, then compare 10%, 25%, and 50% densities against random and magnitude pruning. Test alpha in {0.25,0.5,0.75,1}. Measure immediate post-pruning accuracy, accuracy after a fixed fine-tuning budget, recurrent multiply-adds, and hidden-state explosion or vanishing. Success means better accuracy than magnitude pruning at equal density and faster recovery, while sparse inference realizes the expected reduction in recurrent operations.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built covariance-aware stochastic recurrent-weight pruning with density calibration and strength rescaling in experiment.py. The toy verification confirmed all three quantitative predictions: covariance-estimation RMS error scaled with observed slope -0.579 versus predicted -0.5, calibrated K scaled 4.0x when covariance was multiplied by 4, and alpha=1 preserved total absolute weight to 5.8e-14 relative error. In the tiny RNN comparison, covariance pruning was worse than magnitude pruning at 50% density (0.618 vs 0.895 accuracy), so there is no observed task-performance win in this MVP.

### Mechanism check

- Verdict: Built covariance-aware stochastic recurrent-weight pruning with density calibration and strength rescaling in experiment.py. The toy verification confirmed all three quantitative predictions: covariance-estimation RMS error scaled with observed slope -0.579 versus predicted -0.5, calibrated K scaled 4.0x when covariance was multiplied by 4, and alpha=1 preserved total absolute weight to 5.8e-14 relative error. In the tiny RNN comparison, covariance pruning was worse than magnitude pruning at 50% density (0.618 vs 0.895 accuracy), so there is no observed task-performance win in this MVP.
- Confidence: 9/10
- Limitations: Only one small synthetic sequence-classification task, one-shot pruning, one density, and one covariance-noise setting were tested. No fine-tuning schedule, Copy Memory or sequential MNIST, multiple trained-network seeds, sparse-kernel speed benchmark, or comparison across alpha and density values was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/683/bench_report.json)
- [bench_run.py](https://synthcore.org/code/683/bench_run.py)
- [experiment.py](https://synthcore.org/code/683/experiment.py)
- [report.md](https://synthcore.org/code/683/report.md)
- [report_bench_2026-08-31T214619.md](https://synthcore.org/code/683/report_bench_2026-08-31T214619.md)
- [results.json](https://synthcore.org/code/683/results.json)
- [Download all files as ZIP](https://synthcore.org/download/683)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.