✗ Failed on benchmark 2026

Noise-prune recurrent weights by covariance-aware retention

Implementation & benchmark of arXiv:2608.05464 — Effective pruning of task-trained recurrent neural networks using noisy fluctuations and connection rescaling

Usefulness7/10

Difficulty4/10

Novelty7/10

Regularization Memory Speedup Accuracy Rnn Ssm Pruning Regularization Probability Statistics Linear algebra Dynamical systems

Source paper: Effective pruning of task-trained recurrent neural networks using noisy fluctuations and connection rescaling arXiv:2608.05464 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace magnitude pruning in a trained recurrent network with stochastic pruning probabilities computed from weight magnitudes and the covariance of neuron activities under injected noise. Connections whose endpoints fluctuate in a sign-compatible way receive higher retention probability, while retained weights are rescaled to preserve average recurrent strength. The method uses local weights and activity covariance, avoiding Hessian construction and expensive global saliency optimization.

Formulas

$$p_{ij}=\begin{cases}K w_{ij}\left(C_{ii}+C_{jj}-2C_{ij}\right),&w_{ij}>0\\K|w_{ij}|\left(C_{ii}+C_{jj}+2C_{ij}\right),&w_{ij}<0\end{cases}$$

$$C_{ii}+C_{jj}-2C_{ij}=\operatorname{Var}(x_i-x_j),\qquad C_{ii}+C_{jj}+2C_{ij}=\operatorname{Var}(x_i+x_j)$$

$$m_{ij}\sim\operatorname{Bernoulli}(p_{ij}),\qquad \widetilde w_{ij}=m_{ij}w_{ij}\left(\frac{\sum_{a,b}|w_{ab}|}{\sum_{a,b}m_{ab}|w_{ab}|+\varepsilon}\right)^{\alpha}$$

$$p_{ij}=\min\{1,Ks_{ij}\},\qquad K\ \text{chosen so that}\ \frac{1}{|E|}\sum_{(i,j)\in E}p_{ij}=\rho$$

Mathematical statement

Let x_i(t) be the activity of neuron i under noisy input, let w_{ij} be the directed weight from neuron j to neuron i, and let C be the activity covariance matrix, with C_{ij}=Cov[x_i,x_j] and C_{ii}=Var[x_i]. The paper defines the retention probability p_{ij} using sign-dependent fluctuation energy: for an excitatory weight w_{ij}>0, p_{ij}=K w_{ij}(C_{ii}+C_{jj}-2C_{ij}); for an inhibitory weight w_{ij}<0, p_{ij}=K|w_{ij}|(C_{ii}+C_{jj}+2C_{ij}). K controls overall density. The two covariance expressions equal Var[x_i-x_j] and Var[x_i+x_j], respectively, so they are nonnegative and measure endpoint fluctuations in the sign-compatible direction. In implementation, clip probabilities to [0,1] using p_{ij}=min(1,Ks_{ij}), where s_{ij} is the relevant score. Sample m_{ij}~Bernoulli(p_{ij}) independently and set the pruned weight to zero when m_{ij}=0. A practical layerwise rescaling is w'_{ij}=m_{ij}w_{ij}R^alpha, where R=sum_{a,b}|w_{ab}|/(sum_{a,b}m_{ab}|w_{ab}|+epsilon), epsilon prevents division by zero, and alpha controls rescaling strength. Alpha=1 preserves total absolute weight approximately; alpha<1 tests the paper's observation that weaker empirical rescaling can outperform the theoretically motivated value.

Implementation notes

1. Integration point: apply the method to the recurrent matrix W_hh of an Elman RNN, GRU, or leaky state-space model after ordinary pretraining. It can also be repeated in a prune-and-finetune schedule, but initially use one-shot pruning for a clean comparison. 2. Pseudocode: initialize covariance accumulators S=0 and Q=0. For b=1,...,B, run minibatches with small zero-mean Gaussian recurrent or preactivation noise and collect hidden states H_b in R^{T x N}. Update S by the sum of hidden vectors and Q by the sum of outer products. After M total time points, compute mu=S/M and C=Q/M-mu mu^T. For every W[i,j], compute s[i,j]=W[i,j]*(C[i,i]+C[j,j]-2*C[i,j]) when W[i,j]>0; otherwise compute s[i,j]=abs(W[i,j])*(C[i,i]+C[j,j]+2*C[i,j]). Choose K by bisection so the mean of min(1,K*s[i,j]) equals target density rho. Sample mask[i,j] from Bernoulli(min(1,K*s[i,j])), zero masked weights, and compute R=sum(abs(W))/(sum(abs(mask*W))+epsilon). Set W=mask*W*R^alpha and fine-tune. 3. Quantities supplied by the mathematics are the sign-dependent score, Bernoulli retention rule, and strength rescaling. Estimate C from noisy hidden-state samples; select rho, noise amplitude, alpha, and B empirically. Use separate covariance-estimation and validation batches. Preserve self-connections or enforce a minimum in-degree if pruning destabilizes the recurrent dynamics. 4. First experiment: train 64- or 128-unit RNNs on Copy Memory, adding, and sequential MNIST, then compare 10%, 25%, and 50% densities against random and magnitude pruning. Test alpha in {0.25,0.5,0.75,1}. Measure immediate post-pruning accuracy, accuracy after a fixed fine-tuning budget, recurrent multiply-adds, and hidden-state explosion or vanishing. Success means better accuracy than magnitude pruning at equal density and faster recovery, while sparse inference realizes the expected reduction in recurrent operations.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built covariance-aware stochastic recurrent-weight pruning with density calibration and strength rescaling in experiment.py. The toy verification confirmed all three quantitative predictions: covariance-estimation RMS error scaled with observed slope -0.579 versus predicted -0.5, calibrated K scaled 4.0x when covariance was multiplied by 4, and alpha=1 preserved total absolute weight to 5.8e-14 relative error. In the tiny RNN comparison, covariance pruning was worse than magnitude pruning at 50% density (0.618 vs 0.895 accuracy), so there is no observed task-performance win in this MVP.

Agent confidence: 9/10

Baseline: Magnitude pruning: 0.8950 post-pruning accuracy at recurrent density 0.5; dense accuracy 0.9117.
Idea: Covariance-aware stochastic pruning: 0.6177 post-pruning accuracy at recurrent density 0.5; toy prediction checks passed with observed sampling slope -0.579 versus predicted -0.5, K ratio 4.0, and alpha=1 relative L1 error 5.8e-14.

Limitations: Only one small synthetic sequence-classification task, one-shot pruning, one density, and one covariance-noise setting were tested. No fine-tuning schedule, Copy Memory or sequential MNIST, multiple trained-network seeds, sparse-kernel speed benchmark, or comparison across alpha and density values was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.05464) Sep 2, 2026

Significantly WORSE than the tuned baseline (+442.2%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.002
Idea mean: 0.0108
Effect (Δ): 0.0088 (+442.2%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±205.9%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the dynamics track was tested; no vision, tabular, or sequence tracks were run. The recurrent mask calibration produced an observed density far below the target, so the result is not a clean test of the intended 50% density method and should be debugged before drawing a definitive scientific conclusion. No sparse-kernel inference speed or memory benchmark was measured, and the K scaling prediction was not re-tested on trained-model covariance estimates.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Built a covariance-aware stochastic recurrent-weight pruning intervention for the structurally matched dynamics track using the benchmark rnn_small GRU, with magnitude pruning as the matched baseline, shared learning-rate sweep, equal pretraining/fine-tuning budgets, and 8 paired seeds. The baseline achieved test MSE 0.001991 versus 0.010792 for the idea; the paired permutation test gave delta_mean=+0.008802 and p=0.0081, so the idea was significantly worse. The trained-model mechanism signature was not confirmed: observed sampled density was 0.0876 rather than 0.5 and observed L1 ratio was 0.201 rather than 1.0, indicating an implementation/calibration issue that further weakens any claim of benefit.

Baseline: dynamics/rnn_small magnitude pruning, density target 0.5, best lr=0.003: mean test MSE 0.001990619653952308, std 0.0008054264134080661, per-seed [0.0018116666469722986, 0.0022851978428661823, 0.0010032905265688896, 0.0023619511630386114, 0.001113392529077828, 0.001145130256190896, 0.003139213193207979, 0.003065115073695779]
Idea: covariance-aware stochastic pruning, best tested alpha=0.25, noise=0.03, lr=0.003: mean test MSE 0.010792315122671425, std 0.004284810897028596, per-seed [0.011267757043242455, 0.009967799298465252, 0.012210535816848278, 0.0203185323625803, 0.00917056854814291, 0.011397886089980602, 0.004530716221779585, 0.007474725600332022]
Paired delta mean: idea-baseline = +0.008801695468719117 MSE; p_value=0.0081; verdict=idea worse (significant)

Benchmark result