Effective pruning of task-trained recurrent neural networks using noisy fluctuations and connection rescaling

arXiv:2608.05464 2026 Regularization 1 ideas extracted · analyzed Aug 31, 2026

What the math gives to ML

The paper provides a local, covariance-aware stochastic pruning rule for recurrent connections. Instead of ranking weights by magnitude, it estimates how strongly noisy fluctuations at the source and target neurons make a synapse functionally important, with different covariance combinations for excitatory and inhibitory weights. This is transferable to recurrent and state-space networks because the statistics come from noisy forward passes and require neither Hessians nor global saliency optimization. Stochastic retention followed by controlled rescaling can preserve task performance at a fixed sparse density.

Ideas from this paper

✗ Failed on benchmark 2026

Noise-prune recurrent weights by covariance-aware retention

Replace magnitude pruning in a trained recurrent network with stochastic pruning probabilities computed from weight magnitudes and the covariance of neuron activities under injected noise. Connections whose endpoints fluctuate in a sign-compatible way receive higher retention probability, while retained weights are rescaled to preserve average recurrent strength. The method uses local weights and activity covariance, avoiding Hessian construction and expensive global saliency optimization.

Useful7/10

Difficulty4/10

Novelty7/10

Regularization Memory Speedup Accuracy

Paper: Effective pruning of task-trained recurrent neural networks using noisy fluctuations and connection rescaling arXiv:2608.05464